Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Register

Quick Links


Categories

In this Discussion

› › Automatically generated IP blocklists of various types
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Automatically generated IP blocklists of various types

SplitIceSplitIce Member, Host Rep
edited May 2021 in General

I thought I would share this here as there are many people here who may find lists of network abusers and common bots to be useful in their projects.

We (at X4B) recently decided to undertake a project to modernise our blocklist generation. As part of that work we opted to make the vast majority of our lists open source.

Now you can (from GitHub) fetch lists of many kinds that are:
- CI generated (Github Actions via daily schedule)
- consistently formatted (all lists are newline seperated CIDRs)

Lists currently include:
- TOR Exit Nodes
- VPN / server networks
- Search Engines
- StopForumSpam
- Uptimerobot
- Paypal
- Cloudflare
- Some extras

In case you don't know you use the raw button to get a consistent link to add to pfsense and similar software :)

Pull requests and requests for additional lists are welcome.

If you find these useful please do let me know. I've got a few other ideas for other ideas for similar open source (Open source GeoIP anyone?) work and justification for the effort would be grand.

Also thanks @Meganitrospeed for his work on the VPN list. In many ways it was his request that inspired this specific approach.

Thanked by 13varwww sgheghele mrTom skorous isunbejo dosai lentro chocolateshirt LordSpock Hotmarer AlwaysSkint FAT32 kw0z

Comments

  • sgheghelesgheghele Member

    Thanks for this!

    What would be the reason for blocking UptimeRobot, PayPal, and Cloudflare?

  • SplitIceSplitIce Member, Host Rep

    @sgheghele said:
    Thanks for this!

    What would be the reason for blocking UptimeRobot, PayPal, and Cloudflare?

    Any blacklist can also be a whitelist (or exception list).

    The Paypal list is particularly unique by the way. Web scraping is required for that one :(

    Thanked by 1sgheghele
  • LeviLevi Member

    Interesting, why anyone want to block PayPal?

  • seriesnseriesn Member

    @LTniger said:
    Interesting, why anyone want to block PayPal?

    You block them before they block you.

    Thanked by 5MannDude CConner yoursunny lentro jetchirag
  • jmgcaguiclajmgcaguicla Member

    @seriesn said:

    @LTniger said:
    Interesting, why anyone want to block PayPal?

    You block them before they block you.

    I outsmarted your outsmarting

    image

  • isunbejoisunbejo Member

    duckduckgo ip list ?

  • RazzaRazza Member
    edited May 2021

    @LTniger said:
    Interesting, why anyone want to block PayPal?

    Instead of blocking you could use the ip's for whitelisting.

  • stefemanstefeman Member

    @sgheghele said:
    Thanks for this!

    What would be the reason for blocking UptimeRobot, PayPal, and Cloudflare?

    Spoofed DDoS is one. By spoofing into those ranges, the attack might get through some filters. For example, OVH is known to have global whitelist for some well known ip ranges.

    Thanked by 1sgheghele
  • SplitIceSplitIce Member, Host Rep

    @isunbejo said:
    duckduckgo ip list ?

    I've added a feature request to the internal issue. I don't have any data on their crawler currently however. I'd need to first build a reliable detecter (we generally require IP validation from an authorative source e.g ns1 for the company).

    I'll look into it in the future however.

    Thanked by 1isunbejo
  • eKoeKo Member

    @SplitIce said:

    @isunbejo said:
    duckduckgo ip list ?

    I've added a feature request to the internal issue. I don't have any data on their crawler currently however. I'd need to first build a reliable detecter (we generally require IP validation from an authorative source e.g ns1 for the company).

    I'll look into it in the future however.

    https://help.duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/

    It would be nice to have the iplist's in nginx format aswell ?

    Thanks for the hard work!

  • HotmarerHotmarer Member
    edited June 2021

    Add VPN like NordVPN and others. You can simple enumerate all subdomains to get ip.

    Edit: https://ipinfo.io/AS136787#blocks also need to be blocked

  • HotmarerHotmarer Member

    @eKo said: It would be nice to have the iplist's in nginx format aswell ?

    it would also be great to get a list with ip without classes, only addresses like not 1.1.1.0/24 but 1.1.1.0, .1.1.1.1, 1.1.1.2, 1.1.1.3 etc.

  • bulbasaurbulbasaur Member

    @Hotmarer said: it would also be great to get a list with ip without classes, only addresses like not 1.1.1.0/24 but 1.1.1.0, .1.1.1.1, 1.1.1.2, 1.1.1.3 etc.

    $ python3
>>> import ipaddress
>>> [str(x) for x in ipaddress.ip_network('1.1.1.0/28')]
['1.1.1.0', '1.1.1.1', '1.1.1.2', '1.1.1.3', '1.1.1.4', '1.1.1.5', '1.1.1.6', '1.1.1.7', '1.1.1.8', '1.1.1.9', '1.1.1.10', '1.1.1.11', '1.1.1.12', '1.1.1.13', '1.1.1.14', '1.1.1.15']
  • HotmarerHotmarer Member

    @stevewatson301 said:

    @Hotmarer said: it would also be great to get a list with ip without classes, only addresses like not 1.1.1.0/24 but 1.1.1.0, .1.1.1.1, 1.1.1.2, 1.1.1.3 etc.

    $ python3
>>> import ipaddress
>>> [str(x) for x in ipaddress.ip_network('1.1.1.0/28')]
['1.1.1.0', '1.1.1.1', '1.1.1.2', '1.1.1.3', '1.1.1.4', '1.1.1.5', '1.1.1.6', '1.1.1.7', '1.1.1.8', '1.1.1.9', '1.1.1.10', '1.1.1.11', '1.1.1.12', '1.1.1.13', '1.1.1.14', '1.1.1.15']

    I know, but then I have to convert the list myself, then upload it somewhere to only import addresses to devices which do not support ip addresses with classes

  • AlwaysSkintAlwaysSkint Member
    edited June 2021

    Great idea!

    Here's a couple for you..
    Shodan idiots.
    All those friggin' census/so-called research sites!

    Network class C 185.180.143.0/24 (PT/Portugal/-) has been blocked
    IP addresses that triggered the block
    Mon May 31 10:07:37 2021 (bind) bind triggered by 185.180.143.14 (PT/Portugal/sh-chi-us-gp1-wk103.internet-census.org): 1 in the last 3600 secs
    Mon May 31 13:05:38 2021 (bind) bind triggered by 185.180.143.76 (PT/Portugal/sh-phx-us-gp1-wk109.internet-census.org): 1 in the last 3600 secs
    Mon May 31 16:08:19 2021 (bind) bind triggered by 185.180.143.13 (PT/Portugal/sh-chi-us-gp1-wk102.internet-census.org): 1 in the last 3600 secs
    Mon May 31 19:04:00 2021 (bind) bind triggered by 185.180.143.77 (PT/Portugal/sh-phx-us-gp1-wk110.internet-census.org): 1 in the last 3600 secs
    Mon May 31 22:14:06 2021 (bind) bind triggered by 185.180.143.142 (PT/Portugal/sh-ams-nl-gp1-wk114.internet-census.org): 1 in the last 3600 secs

  • SplitIceSplitIce Member, Host Rep

    @eKo said:

    @SplitIce said:

    @isunbejo said:
    duckduckgo ip list ?

    I've added a feature request to the internal issue. I don't have any data on their crawler currently however. I'd need to first build a reliable detecter (we generally require IP validation from an authorative source e.g ns1 for the company).

    I'll look into it in the future however.

    https://help.duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/

    It would be nice to have the iplist's in nginx format aswell ?

    Thanks for the hard work!

    Ugly to scrape but not a horrible idea.

    Honestly I'm torn between submitting the data from detection and fragile web scraping.

    For now I'll say. PR welcome. See lists_paypal for an example of scraping.

    @Hotmarer said:

    @eKo said: It would be nice to have the iplist's in nginx format aswell ?

    it would also be great to get a list with ip without classes, only addresses like not 1.1.1.0/24 but 1.1.1.0, .1.1.1.1, 1.1.1.2, 1.1.1.3 etc.

    For large frequently updating lists like lists_vpn that would be too much I think. Part of the reason I'm going with subnets is to keep the resource usage for GitHub reasonable.

    There are plenty of easy to work with tools for making /32 IP Lists from a CIDR List. I also believe that's within the capability of most people, but if there is a good argument to be made I'm happy to hear it.

    @eKo said: It would be nice to have the iplist's in nginx format aswell ?

    Not a bad idea. awk could do it quite easily. PRs accross the repos welcome.

    @Hotmarer said: Add VPN like NordVPN and others.

    PR welcome. see this commit for your provided ASN for an example.

    https://github.com/X4BNet/lists_vpn/commit/3824a35f9be687c37d3fc7ad24a0ffd6029b65e4

    @Hotmarer said:

    You can simple enumerate all subdomains to get ip.

    Processing RDNS db's is not particularly fast and the DB's I've seen are massive to download. Not something that lends itself to CI processing.

    While there are sql dbs some people make of all RDNS results for the IPv4 space they arent easy to process in reasonable CI time.

    PR for CI scripts processing RDNS is welcome if it's:
    a) Necessary to cover a particular case
    b) Reasonable to process (time)
    c) Data sourced from a trustworthy source

  • SplitIceSplitIce Member, Host Rep

    @Hotmarer said: I know, but then I have to convert the list myself, then upload it somewhere to only import addresses to devices which do not support ip addresses with classes

    I mean you could use a CI script to automate that...

    If you have a valid case PR quality CI scripts and make a compelling case. Of course you only need to make the case if you want to merge upstream, else you can run your fork. Ain't git great.

  • SplitIceSplitIce Member, Host Rep
    edited June 2021

    @Hotmarer for note the lists_vpn repository currently covers 140,385,323 IPs. Ballpark estimation of an individual IP list says that would be an at-least 2GB. Running the risk of not even being commitable to Github...

  • bulbasaurbulbasaur Member

    @SplitIce said: Running the risk of not even being commitable to Github...

    Commit the scripts and have Github actions upload to the releases function?

  • SplitIceSplitIce Member, Host Rep
    edited June 2021

    @stevewatson301 said: Commit the scripts and have Github actions upload to the releases function?

    Personally I think that would fall under rule one of free services (e.g Github's free open source repositories). Don't be a Dick.

    Anyway

    Each file included in a release must be under 2 GB. There is no limit on the total size of a release, nor bandwidth usage.

    Not to mention it wouldnt be much use, you would be swapping the need to process the list for the need to form download URLs for the release assets.

  • HotmarerHotmarer Member

    @SplitIce said: Processing RDNS db's is not particularly fast and the DB's I've seen are massive to download. Not something that lends itself to CI processing.

    While there are sql dbs some people make of all RDNS results for the IPv4 space they arent easy to process in reasonable CI time.

    PR for CI scripts processing RDNS is welcome if it's:
    a) Necessary to cover a particular case
    b) Reasonable to process (time)
    c) Data sourced from a trustworthy source

    All you have to do is query subdomains. No needed RDNS here.
    alX.nordvpn.com - Albanian servers

  • HotmarerHotmarer Member

    @SplitIce said:
    @Hotmarer for note the lists_vpn repository currently covers 140,385,323 IPs. Ballpark estimation of an individual IP list says that would be an at-least 2GB. Running the risk of not even being commitable to Github...

    You're right, I didn't think it through.

Sign In or Register to comment.