New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Maxmind's API and a bit of simple code
Do you need everything Blocked offers or just country block/proxy block/vpn block?
Why some folks strictly need to know your real data, IP address, country etc if visiter don't want to share it? Where the privacy rights?
@jenkki why should a site let you visit if it doesn't know where you are from?
block/proxy block/vpn block
I want to block/proxy,VPN
They want $5 for 1k lookups , they are even much more expensive than blocked.com
i need 100k/day lookups
They all just earn money on your spymania
@black's http://getipintel.net maybe?
Thanks Ishaq for the mention.
@flym Feel free to email me if you want to discuss requirements.
It is better cache the results from blocked.com.
getipintel.net estimation is useless. There are so many false positives in 0.95+ zone.
Can you PM me a list of those IPs if you still have it? Also we don't recommend you take automated action against values < 0.99. The different flags also matter because different datasets are included. False positives are more likely when you include bad IP detection which includes compromised systems. Generally, these compromised systems are from legitimate ISPs. If the ISP uses dynamic IP allocation then someone might get a dirty IP from that ISP's IP pool. The infected user would most likely complain as well because they wouldn't know their system is compromised. For proxy / VPN detection only we recommend you use flags=m option. If that doesn't catch enough proxy / VPNs then use flags=b.
I talked with getipintel.net too
they charge too much too , it will be few times expensive more than blocked.com
is there anyway i can do it by hire a programmer?
Sorry I couldn't help. I'm not 100% sure with how blocked does their proxy detection but I think they use blacklists. The way we implement proxy / VPN detection goes beyond a simple blacklist lookup. It actually tries to infer a proxy / VPN score based on machine learning & probability theory techniques and it uses very large datasets to do so. Right now there's 690+ million active records. The score is actually computed in real time with that information and our algorithm so it's not cheap to operate.
Mainly, to do start with this sort of thing, you need information. Blacklisting is the easiest way. I've seen people just try to connect to the IP on port 80, 8080, 3128. If a connection is successful then they say it's a proxy. You can also look in the HTTP headers for HTTP_X_FORWARDED_FOR, HTTP_FORWARDED_FOR, HTTP_X_FORWARDED, etc. Downside is that most proxies don't set those headers and port "scanning" isn't a very accurate way of determining if it's a proxy / VPN.
https://medium.com/@ValdikSS/detecting-vpn-and-its-configuration-and-proxy-users-on-the-server-side-1bcc59742413#.590wxitkb is a good article if you just want to deal with OpenVPN connections, however the article was written awhile ago I'm not sure if it's still true or not.
Hopefully this information is enough to get you started.
How will be performance of server to read blacklist from Database
It'll be pretty good. If you have single IP addresses, the keys would be hashed so it's very fast. If you get them in CIDR format you can just use ramdisk or something and throw the entire file in RAM. You can do some more optimizations as well but it should be more than enough to get you started.
Do you know where i can buy list of Proxies ?
I do not. I remember someone posting some ASNs awhile ago but I can't find it anymore and it's really old.
Meh, it is a waste of time, we already had that conversation a year+ ago and your reply was "Unfortunately, I can't do much ...".
Unfortunately I'm a customer of small (local) residential ISP (~10k customers) providing mainly static IPs. And most of their IPs come out as 0.95+. Every time when I have to signup somewhere (where they use getipintel) I have to waste time to explain that I'm not using proxy/VPN etc.
My IP is not listed on any SPAM-list nor on any other bad-list. The last time I got "infected" was in Win 98 era.
flags=m may be good, but it is not me that choose how to query your database, I'm just a victim of your "machine learning & probability theory techniques".
@flym - try javascript peer connection trick (ofc obfuscate the js and the data). Then based on the result check a static blacklist(s) in your database. Then if really needed use external checker (cache the result for few days/weeks). Make sure you are checking only once per IP and not per request (yes, I've seen this too). It really depends on your needs. Memcache is really fast choice, but it uses extra memory.
A couple of things to reduce your lookups:
Your budget might still be too low but the good thing is that over time, your database builds up and your cost will reduce.
Here is our implementation from when we did this (since then we just removed the possibility to buy anything from the site):
https://gist.github.com/jameshadley/5a749b8145da302197b3ae339220e0d2
Obviously you'll need to adapt it a bit.
Like others have said, I think you'll decide this is a waste of time at some point.
With some query caching I'm sure you won't need 100k queries per day.
A lot of things have changed since then. The datasets and the code aren't the same as it was a year ago. I also have code that lets me force output to be not higher than some value which is currently set at 0.95 (again, we don't recommend people take action on values < 0.99 so it should be ok). Believe it or not, we're not serving nearly enough queries for you to run into many different websites that uses our API so some sites aren't using our implementation of proxy / VPN detection. If you want me to take a look at the issue, feel free to PM me. If not, that's ok too.
@Black, have you published any info / code / data about how your proxy detection works? I mean with more detail than "oh we use machine learning techniques". I'm more interested in the technical side of how it's done, than in actually using it.
Also, if you assign a probability score of 95% to address X, does that mean that a randomly chosen address would have about 5% likelihood of having that high a score or higher? Or does it take into account some presumed (and hopefully low) prior probability that a random address is actually a proxy?
Thanks!
Some more information is available in the FAQs, on the website it's called dynamic checks.
Assuming the assumptions are met when you're using the API:
No. The number of proxies in the IP address space is a very small percentage. If you mean randomly chosen in a pool of known proxies, then it's not really that either.
Suppose 100 IPs generate a score of 0.99. That means 1 out of those 100 IPs is a false positive. Suppose 100 IPs generate a score of 0.98, then that means 2 out of every 100 IPs is a false positive. This is based on the information the system has at that moment. So one might say f = (1-s)*100 where f is the # of false positives and s is the score, assuming the law of large numbers. In our testing, it was more like f = 1.07 ( 1 - s ) * 100 when s < ~0.99 so the lower the s, the rate of false positives increase at a rate of around 1.07 for s < 0.99. This means that if there's 100 IPs and all of them generated 0.95, you'd only expect 5 false positives but real data testing shows the number is > 5. Additionally, for values > 0.99 like 0.999 you'd expect to see 1 false positive out of 1000, but no false positives were observed during our tests. This is why I recommend to not take automated action against values < 0.99. If you do set it at 0.95, you should manually review that score and what the person is doing so you can decide for yourself or you should have tested the system thoroughly. Generally speaking, you should test the system thoroughly regardless and set your value(s) to take action on. You can always set it to take automated action on higher values but manually review lower values, it's up to you. Disclaimer: I didn't test for values < 0.85 because I think the score at that point is not useful so the function of false positives might not even be linear. I've spent a lot of time tweaking the algorithm to give good results for > = 0.99 but it's hard to quantity the overall positive effect since the dataset changes very often so please test it yourself.
I should also mention that this is all from my personal testing, experience, and feedback from users. It's not possible for to obtain every active proxy IP and test the system to give definitive values.
I'm not exactly sure I understand the question but the system presumes it's not a proxy until shown otherwise.
I've had no problems with @black's service, I use it as a precaution in my WHMCS installation along with MaxMind. (I use flags=b)
Thanks again @black, my fraud rates are at an all time low
Thanks for the kind words, I'm glad you find it useful.