Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


New project: WHOIS tool with JSON API
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

New project: WHOIS tool with JSON API

joepie91joepie91 Member, Patron Provider
edited September 2012 in General

For those that have a use for it: two days ago or so I wrote a Python module to retrieve and parse WHOIS data, and today I've turned it into an actual web service including JSON API: http://whois.cryto.net

There is no rate-limiting on either HTML or JSON requests, but please be considerate - I'm still working on the spreading out of queries across multiple servers, and WHOIS servers have a tendency of blacklisting IPs that make a lot of requests. For now, all retrieved WHOIS data is indefinitely cached in a local database, so it won't ever be updated. This will change when I get the distribution part done, there will then be an option to refresh stale data.

If you're curious, the web service is written using Flask, served by lighttpd via fastcgi. WHOIS data is stored in mongodb. Source code for this will be released soon.

Currently no domain holder details are parsed yet, as this is a bit more complicated to implement - this will be done at a later stage.

If it doesn't recognize information on certain domains, then feel free to either post them here or PM me so I can improve the parser. Parsing WHOIS data is a messy business :)

If you like it or find it useful, then please consider donating :)

Comments

  • InfinityInfinity Member, Host Rep

    Looks awesome, once the domain holder details etc. come I will definitely switch to using it.

  • joepie91joepie91 Member, Patron Provider

    Another thing I want to add: if you want to use the WHOIS parsing library on your own machine/server without relying on the web service, just pip install pythonwhois and then import pythonwhois; pythonwhois.whois("google.com") - I've added the module to the PyPi repository :)

    Thanked by 2Amfy lbft
  • I love it! nice and simple

  • VPNshVPNsh Member, Host Rep

    Brilliant :). Gonna keep this bookmarked and use it as my main WHOIS search tool :)

  • RaymiiRaymii Member
    edited September 2012

    Looks very nice!

  • joepie91joepie91 Member, Patron Provider

    Updated it - it doesn't parse the registrant details yet, but it now shows the raw WHOIS output if it's available, and provides it via the JSON API as well.

  • Nice work. Flask is awesome and mongodb too :)

  • Looks great and quick to!

    Also the extra amazingness, Registrar: STORMVZ
    Love seeing that :P

  • Great :)
    But you should add something like, no need to put "www"

  • joepie91joepie91 Member, Patron Provider

    But you should add something like, no need to put "www"

    What do you mean? 'www' isn't part of a domain name to start with, so it makes no sense to look it up.

  • Yes, but some mere visitor will add the www for the domain name

  • joepie91joepie91 Member, Patron Provider

    @ErawanArifNugroho said: Yes, but some mere visitor will add the www for the domain name

    Honestly, if someone tries to add www to a whois query, I'm not sure they should be using the tool in the first place... I can't just strip it either, since something like www.co.uk will go wrong then.

  • @joepie91 said: I'm not sure they should be using the tool in the first place...

    Agreed. And very nice tool.

  • bnmklbnmkl Member
    edited September 2012

    Great! So clean.
    Would you be up for adding URL-rewriting @joepie91? I like to be able to just prepend to the URL whilst actually viewing the domain in question.

    http://whois.cryto.net/lowendbox.com
    http://whois.cryto.net/http://www.lowendbox.com/

    Maybe a forwarded short sub-domain too? w.cryto.net

  • joepie91joepie91 Member, Patron Provider

    @bnmkl said: Great! So clean.

    Would you be up for adding URL-rewriting @joepie91? I like to be able to just prepend to the URL whilst actually viewing the domain in question.

    http://whois.cryto.net/http://www.lowendbox.com/

    Maybe a forwarded short sub-domain too? w.cryto.net

    Whoops, completely missed your reply.

    It may be possible to add URL rewriting like that - but this depends on how flexible Flask is in its routing. I'll have to read up on that.

    A short subdomain should be doable as well - I'll look into that soon.

    Thanked by 1bnmkl
  • How often does it update? Just checked my other domain and it's been a week or so since I changed NS but details still show old

  • joepie91joepie91 Member, Patron Provider
    edited September 2012

    @StormVZ said: How often does it update? Just checked my other domain and it's been a week or so since I changed NS but details still show old

    Right now it doesn't update at all yet. I will at some point add a force update option as well as an expiry date, but I first want to get an idea of how many queries are made, and how they are distributed across WHOIS servers, as well as setting it up to distribute queries over multiple VPSes. That way I can make it work reliably without the risk of getting blacklisted.

    All raw WHOIS data is stored, so I can figure out what servers received how many requests, after the fact.

  • .co domains aren't working for me. Love the simplicity of this, great work!

  • joepie91joepie91 Member, Patron Provider

    @NickBudi said: .co domains aren't working for me. Love the simplicity of this, great work!

    Hmm, can you give me a few domains that you tried to look up?

  • Buting58Buting58 Member
    edited September 2012

    @joepie91 said: Hmm, can you give me a few domains that you tried to look up?

    I tried go.com 5mag.co nwc.co

Sign In or Register to comment.