Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In with OpenID
Advertise on LowEndTalk.com

In this Discussion

New project: WHOIS tool with JSON API

New project: WHOIS tool with JSON API

joepie91joepie91 Member
edited September 2012 in General

For those that have a use for it: two days ago or so I wrote a Python module to retrieve and parse WHOIS data, and today I've turned it into an actual web service including JSON API: http://whois.cryto.net

There is no rate-limiting on either HTML or JSON requests, but please be considerate - I'm still working on the spreading out of queries across multiple servers, and WHOIS servers have a tendency of blacklisting IPs that make a lot of requests. For now, all retrieved WHOIS data is indefinitely cached in a local database, so it won't ever be updated. This will change when I get the distribution part done, there will then be an option to refresh stale data.

If you're curious, the web service is written using Flask, served by lighttpd via fastcgi. WHOIS data is stored in mongodb. Source code for this will be released soon.

Currently no domain holder details are parsed yet, as this is a bit more complicated to implement - this will be done at a later stage.

If it doesn't recognize information on certain domains, then feel free to either post them here or PM me so I can improve the parser. Parsing WHOIS data is a messy business :)

If you like it or find it useful, then please consider donating :)

Appreciate my posts/software/guides? Donate (PayPal/Flattr/Bitcoin): http://cryto.net/~joepie91/donate.html | irc.freenode.net #lowendbox

Comments

  • InfinityInfinity Retired Staff

    Looks awesome, once the domain holder details etc. come I will definitely switch to using it.

    我是一个巨魔 (;

  • Another thing I want to add: if you want to use the WHOIS parsing library on your own machine/server without relying on the web service, just pip install pythonwhois and then import pythonwhois; pythonwhois.whois("google.com") - I've added the module to the PyPi repository :)

    Appreciate my posts/software/guides? Donate (PayPal/Flattr/Bitcoin): http://cryto.net/~joepie91/donate.html | irc.freenode.net #lowendbox

    Thanked by 2Amfy lbft
  • I love it! nice and simple

  • Brilliant :). Gonna keep this bookmarked and use it as my main WHOIS search tool :)

    VPN.sh - Secure and affordable VPN services

  • RaymiiRaymii Member
    edited September 2012

    Looks very nice!

    Quis custodiet ipsos custodes?
    https://raymii.org - https://cipherli.st
  • Updated it - it doesn't parse the registrant details yet, but it now shows the raw WHOIS output if it's available, and provides it via the JSON API as well.

    Appreciate my posts/software/guides? Donate (PayPal/Flattr/Bitcoin): http://cryto.net/~joepie91/donate.html | irc.freenode.net #lowendbox

  • Nice work. Flask is awesome and mongodb too :)

    I use http://tuxlite.com to configure all my VPSes and I love it!

  • Looks great and quick to!

    Also the extra amazingness, Registrar: STORMVZ Love seeing that :P

    Patrick ~ INIZ Rep

    | Inactive
  • Great :) But you should add something like, no need to put "www"

    My blog | Server Uptime | I'm not working for any providers in here, all my comments just my own opinion.image
  • @ErawanArifNugroho said: Great :) But you should add something like, no need to put "www"

    What do you mean? 'www' isn't part of a domain name to start with, so it makes no sense to look it up.

    Appreciate my posts/software/guides? Donate (PayPal/Flattr/Bitcoin): http://cryto.net/~joepie91/donate.html | irc.freenode.net #lowendbox

  • Yes, but some mere visitor will add the www for the domain name

    My blog | Server Uptime | I'm not working for any providers in here, all my comments just my own opinion.image
  • @ErawanArifNugroho said: Yes, but some mere visitor will add the www for the domain name

    Honestly, if someone tries to add www to a whois query, I'm not sure they should be using the tool in the first place... I can't just strip it either, since something like www.co.uk will go wrong then.

    Appreciate my posts/software/guides? Donate (PayPal/Flattr/Bitcoin): http://cryto.net/~joepie91/donate.html | irc.freenode.net #lowendbox

  • @joepie91 said: I'm not sure they should be using the tool in the first place...

    Agreed. And very nice tool.

  • bnmklbnmkl Member
    edited September 2012

    Great! So clean. Would you be up for adding URL-rewriting @joepie91? I like to be able to just prepend to the URL whilst actually viewing the domain in question.

    http://whois.cryto.net/lowendbox.com http://whois.cryto.net/http://www.lowendbox.com/

    Maybe a forwarded short sub-domain too? w.cryto.net

    -

  • @bnmkl said: Great! So clean. Would you be up for adding URL-rewriting @joepie91? I like to be able to just prepend to the URL whilst actually viewing the domain in question.

    http://whois.cryto.net/lowendbox.com http://whois.cryto.net/http://www.lowendbox.com/

    Maybe a forwarded short sub-domain too? w.cryto.net

    Whoops, completely missed your reply.

    It may be possible to add URL rewriting like that - but this depends on how flexible Flask is in its routing. I'll have to read up on that.

    A short subdomain should be doable as well - I'll look into that soon.

    Appreciate my posts/software/guides? Donate (PayPal/Flattr/Bitcoin): http://cryto.net/~joepie91/donate.html | irc.freenode.net #lowendbox

    Thanked by 1bnmkl
  • How often does it update? Just checked my other domain and it's been a week or so since I changed NS but details still show old

    Patrick ~ INIZ Rep

    | Inactive
  • joepie91joepie91 Member
    edited September 2012

    @StormVZ said: How often does it update? Just checked my other domain and it's been a week or so since I changed NS but details still show old

    Right now it doesn't update at all yet. I will at some point add a force update option as well as an expiry date, but I first want to get an idea of how many queries are made, and how they are distributed across WHOIS servers, as well as setting it up to distribute queries over multiple VPSes. That way I can make it work reliably without the risk of getting blacklisted.

    All raw WHOIS data is stored, so I can figure out what servers received how many requests, after the fact.

    Appreciate my posts/software/guides? Donate (PayPal/Flattr/Bitcoin): http://cryto.net/~joepie91/donate.html | irc.freenode.net #lowendbox

  • .co domains aren't working for me. Love the simplicity of this, great work!

  • @NickBudi said: .co domains aren't working for me. Love the simplicity of this, great work!

    Hmm, can you give me a few domains that you tried to look up?

    Appreciate my posts/software/guides? Donate (PayPal/Flattr/Bitcoin): http://cryto.net/~joepie91/donate.html | irc.freenode.net #lowendbox

  • NickBudiNickBudi Member
    edited September 2012

    @joepie91 said: Hmm, can you give me a few domains that you tried to look up?

    I tried go.com 5mag.co nwc.co

Sign In or Register to comment.