How to limit the connection speed per IP with nginx?

Amitz

Dear fellow LET'ers,

I would need a quick kick in the butt and enlightment:
Is there a way to limit the connection speed per IP to the webserver to e.g. 2 Mbps? Not only per connection, but for all connections from an external IP in total? So that no visitor will be able to consume more than X Mbps? Can this be achieved with nginx or maybe via the firewall (have IPTables/CSF installed on the server)?

Thank you very much for any hint!
I am really a bit clueless at the moment and would appreciate a helping hand!

Kind regards
Amitz

Corey

@Amitz said: Dear fellow LET'ers,

I would need a quick kick in the butt and enlightment:

Is there a way to limit the connection speed per IP to the webserver to e.g. 2 Mbps? Not only per connection, but for all connections from an external IP in total? So that no visitor will be able to consume more than X Mbps? Can this be achieved with nginx or maybe via the firewall (have IPTables/CSF installed on the server)?

Thank you very much for any hint!

I am really a bit clueless at the moment and would appreciate a helping hand!

Kind regards

Amitz

As far as your question, idk... as far as 2mbit goes.... why so low? It's getting more common for people to have 10mbit at their homes.

Amitz

It's a gallery site. Every image is around 500 KB in size. 2mbit would be absolutely enough to load the images in decent time. Right now, I have some people on fat pipes that consume up to 10 mbit alone which is causing an unnecessary load on the server and too much bandwidth being consumed. I would offer a better experience to all visitors if I limit some of them.

Corey

@Amitz said: It's a gallery site. Every image is around 500 KB in size. 2mbit would be absolutely enough to load the images in decent time. Right now, I have some people on fat pipes that consume up to 10 mbit alone which is causing an unnecessary load on the server and too much bandwidth being consumed. I would offer a better experience to all visitors if I limit some of them.

10mbit is not a fat pipe, and if your disks can't handle 10mbit/s you need to get a better server - or optimize something.

Spencer

Hmm, I know lighttpd can do this with ease as it is built right into it, but as far as nginx im not sure.

Amitz

@Corey said: 10mbit is not a fat pipe, and if your disks can't handle 10mbit/s you need to get a better server - or optimize something.

Jesus... I get your point.
But imagine 10 visitors consuming 10 mbit each at the same time. No more bandwidth for the rest and there are always around 100 visitors online all day long.

@Spencer: Unfortunately, I have to stick with the current configuration and cannot switch to lighttpd.

Cirium

Try (inside of nginx config)

limit_rate 1280k;

1280kilobyte = 10mbit

Amitz

@Cirium said: Try (inside of nginx config)
limit_rate 1280k;
1280kilobyte = 10mbit

That was my first idea, too. But this limits the speed per connection and not per IP unfortunately. So you could still max out the bandwidth by downloading 10 images at once.

I already thought of a combination:
Limit the speed per connection with nginx (limit_rate) and then limit the number of connections per IP with CSF. That would be possible, but sounds somehow awkward to me.

nunim

http://wiki.nginx.org/NginxHttpLimitZoneModule

I think the best bet would probably be using a firewall for rate limiting.

sleddog

@Corey said: 10mbit is not a fat pipe, and if your disks can't handle 10mbit/s you need to get a better server - or optimize something.

You should've stopped at "idk"

"Fat" is a relative term. 10mbit's may not be fat for you. For me it's Damned Fat.

Amitz

Stop beating @Corey, guys! I am sure he meant that my server should easily handle 10 mbit without higher load but he did not take into consideration that there could be multiple people at the same time with that amount of bandwidth...

darknessends

Can u show me ur website, I think they are copying all the images from your site. Else a visitor will easily get loaded a set of 100 thumbnails and than will open each of those who he likes giving enough time for other visitors to complete similar requests.
So if someone is consistently using a major chunk of bandwidth he is probably trying to get all the images from your website.

darknessends

Also check this too
http://www.zoominternet.net/~lazydog/iptables-tutorial.html#HASHLIMITMATCH

Amitz

@darknessends: You are absolutely right. I had this problem before. But since I have some common "webcopier" and "offline browser" filters in place and do not allow more than 150 TCP connections to port 80 via CSF, this problem is mostly gone. Hotlinking is also blocked as good as technically possible.

I would prefer not to post the gallery URL here. It is an adult site and I am only responsible for the technical aspects, not the content.

Amitz

So I could set

limit_rate 200k;

in my nginx config and set

CONNLIMIT=10
# Connection Limit Protection. This option configures iptables to offer more
# protection from DOS attacks against specific ports. It can also be used as a
# way to simply limit resource usage by IP address to specific server services.
# This option limits the number of concurrent new connections per IP address
# that can be made to specific ports
#
# This feature does not work on servers that do not have the iptables module
# xt_connlimit loaded. Typically, this will be with MONOLITHIC kernels. VPS
# server admins should check with their VPS host provider that the iptables
# module is included
#
# For further information and syntax refer to the Connection Limit Protection
# section of the csf readme.txt
#
# Note: Run /etc/csf/csftest.pl to check whether this option will function on
# this server

in CSF. Right? Or am I missing something here? Does anyone know which types of connection are covered by that CSF option? If all type of connections like TIME_WAIT & Co. are taken into account, then 10 connections per IP will not be enough, I guess.

pubcrawler

Good post to get up here.

Nginx rate limiting works very good. Iptables works well also, but those limits are universal and tend to be forgotten by the admin in the future. Tad problematic place to control something like this, but you can do it.

limit_proxy_traffic_zone ZONE_INDEX LIMIT_RATE LIMIT_RATE_AFTER

You should be able to set a universal bandwidth cap by placing up high in your nginx.conf file:
limit_rate 200k;

Typically since server might serve hordes of sites, that would be put in the site specific config file. General idea on the syntax and issue is using zones in Nginx:

limit_proxy_traffic_zone ZONE_INDEX LIMIT_RATE LIMIT_RATE_AFTER

ZONE_INDEX is zone id
LIMIT_RATE is bandwidth, same as limit_rate from nginx configuration.
LIMIT_RATE_AFTER is amount of data downloaded after which limit rate will be used. Similar to limit_rate_after.
Due to nginx configuration being defunct in this matter, third parameter (LIMIT_RATE_AFTER) is now unused. You should set it to zero.

I reserve use of iptables to "security" issues mainly. Hard blocking and things that everyone/everything on the server benefits from.

sleddog

@Amitz said: Stop beating @Corey, guys! I am sure he meant that my server should easily handle 10 mbit without higher load but he did not take into consideration that there could be multiple people at the same time with that amount of bandwidth...

For which he deserves beating

Amitz

@pubcrawler:
In that special case, there is only one website hosted on the server and it will be the only one forever. So I do not have to take the need of other sites into account.
If I understood it right (I love nginx for its benefits but come from the Apache side), then I can only set a bandwidth cap PER CONNECTION with the directives that you have posted. How do I prevent a visitor from using multiple connections to easily circumvent this limit? And let's be honest: Everyone opens multiple images at once in his browser when visiting an adult gallery. It a "hands free" thing, I guess... ;-)

@sleddog:
Yeah, but not that hard! ;-)

ramnet

For what you want to do I would suggest a combination of limiting connections per IP and bandwidth per connection.

in your server section

limit_rate 128k;
limit_zone one $binary_remote_addr 10m;

in location section:

limit_conn one 10;

For an image hosting site these kinds of settings should be sufficient (1mbit times up to 10 connections at once).

Because of the way the http works opening multiple connections to load a webpage (most web browsers open somewhere between 4 and 8 connections typically), this is the best way to do it so things don't break.

Also, I would strongly suggest you stop limiting connections in CSF at the firewall level. Doing it at the web server level is much less likely to break things for legitimate clients. The last thing you want is page load errors because someones web browser tried to open too many connections (ie, somebody opening multiple windows/tabs at once, or viewing a gallery type page with lots of resources on them, or whatever).

Amitz

@ramnet
Thank you very much. So your example would allow 10 connections per IP with 1 Mbit each - Did I understand this right? Please excuse me for appearing dumb...

ramnet

@Amitz said: So your example would allow 10 connections per IP with 1 Mbit each - Did I understand this right?

Yes, that's correct.

Amitz

I really have to apologize, it was a hard long day, so please bear with me because your help is much appreciated: Wouldn't that still enable people "sucking" 10mbit at once without problem?

ramnet

@Amitz said: I really have to apologize, it was a hard long day, so please bear with me because your help is much appreciated: Wouldn't that still enable people "sucking" 10mbit at once without problem?

You can always lower it to eg: 64k (which would be 5mbit) or even 32k (2.5 mbit) or 16k (1.25 mbit).

I would strongly advise against reducing the number of connections below 10 however limiting bandwidth per connection to less is perfectly fine.

I wouldn't lower it below 16k though as much below that will be very noticeably slow for even small files loading.

Amitz

Okay, fully understood now. ;-)
I am very thankful for your advice and expertise!

Nexus

GOOGLE IS YOUR BEST FRIEND

Amitz

Wow, you were really so bored that you had to come up with that stupid comment? Shame on you!
I googled, rest assured. I just did not understand without help. ;-)

pubcrawler

Good post @ramnet.

@Nexus, Googling it isn't always clear, apparent or understandable for many issues by many folks. It's all about knowing exactly what you need to search for and by that time you already know whatever, so little to no use for searching for it.

There's power in folks sharing collective knowledge like this. Far more efficient and lots of opportunity for learning and teaching.

Amitz

@pubcrawler: Many thanks also to you and to everyone who has been helpful in this thread! :-)

Amitz

To add something useful:
nginx complained while restarting that the "limit_zone" directive is deprecated and suggests to use "limit_conn_zone" instead.

So it is:
limit_conn_zone $binary_remote_addr zone=one:10m;
instead of:
limit_zone one $binary_remote_addr 10m;

pubcrawler

That's one of my pet peeves about Nginx is the deprecated syntax and commands that use to work, but now do not. Has to create tons of confusion and issues for folks getting started with new stuff in Nginx.

Remember, after changing your config files to always test before killing Nginx or you will have an outage while fixing your config:
/etc/init.d/nginx configtest

connercg

I ran across these two links, I was curious as I will need to do the same thing when a photo repository project launches later next month. The second seems to be a familiar name...

tumblr.jonthornton.com/post/641219929/rate-limiting-with-nginx

kbeezie.com/nginx-protection/