New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
How many IP's can I put into ipset before it starts to bog down the server?
I'm using a Xeon E3 server. I need to add a blacklist that consists of about 25,000 elements. The elements are a mix of individual IP's and subnets so I will be using hash:net ipset list type.
Wondering if that many elements will bog down the throughput and/or increase latency. This is an OpenVZ server with around 100 VPS servers on it. About 4TB of bandwidth usage a month. I have state enabled with related, established iptables entry. So it's just new connections that will go through the ipset iptables entry.
There aren't a lot of benchmarks around. The ones I have found seem to indicate it should be ok. Curious if anyone is doing something similar.
Comments
ipset will not have any issues with 25,000 elements.
We have ipset rules with more than 100,000 IP addresses or ranges and never had any problems.
Yea, that question was one that I was having trepidation about when I was initially setting up ipsets for country blocking.
That ipset for geo blocks has 26,106 elements (hash:net) and never skips a beat.
Been running that way now for about 1.5 years on 5 different servers.
Edit: I also have 1 other ipset of about 3,000 elements on the same servers.
I have been running a couple ipsets with a few thousand IP's for years now. So I know that is solid. I am now planning to increase that 10 fold. Kind of hard to imagine how tens of thousands of IP's can be checked almost instantaneously without delaying/dropping packets. That seems to be exactly what ipsets is designed for with that hash scheme.
I suppose the real question is not so much to do with the size of the set alone but the size of the set vs the frequency and volume of the new connections, if it only had to run a few hash every few seconds it would be fine but your right, lack of real world benches makes it a bit of a suck it and see situation.
Could you perhaps simulate loads until it starts to cause iowait?
It should be able to do a lot more than just a few hash checks a second. Will find out I guess.
Ipset scales well, you will need a large amount of memory for large sets of course though. Alot more than say xt_geoip.
100,000 is doable, I am skeptical about 1,000,000 entries without tweaking.
25,000 elements on hash:net type takes up about 350k. So memory usage is minimal with those numbers. Assuming that scales linearly, 1Mil elements shouldn't use more than about 15MB. Yea it would probably slow things down considerably. Can't find any benchmarks on modern hardware/kernels to find out where the limits are.
Size of a set is rarely a direct problem, it's the frequency of lookups against that set that will determine when things will crumble. You have to understand your traffic patterns and test against that.
@sman said:
That really depends on the hash size and desired chain length. You can make the hash table any size you want, smaller just increases the chain length. Total memory varies. A smaller hash size results in less memory usage, but more CPU usage due to more chains needing to be explored (and each time a de-referenced pointer jump).
Additionally If you keep your sets less than 4MB (possibly less, depending on memory fragmentation) you also benifit from kmalloc rather than vmalloc which can easily improve performance (ipset is not particularly CPU intensive, more memory).
I just let it set the hash size automatically. Entering 25,000 elements increases the hash size from the default 1024 to 8192 bytes.
hash size is not measured in bytes, its a modulus.
Therefore your chain length is at a minimum 4, likely 5-10 depending on has balance.
Whereas with a smaller number, say ~<1024 your chain length is likely 1-2
In this case, if you not want use xtable you can use aggregate network inside ipset.
Best is to create a download cron script that take raw data from http://www.ipdeny.com/ipblocks/