All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Dealing with DoS when you're the source
I'm getting to be at my wit's end here. We're having issues with DoS attacks coming from our own nodes. This is the source of 100% of our downtime.
It's not in our budget right now to upgrade our router or buy an appliance that mitigates this. Our datacenter is completely unwilling to help us with this issue (guess they don't like our money after all.)
It's hard to research this issue. Most of the things on Google seem to be regarding when you're the target, not when you're the source.
Here's some things i've tried so far:
-that old iptables block we've been seeing since 2003. Not a good solution, as it limits all connections.
-floodmon. This caused an immediate kernel panic when starting it.
-The script here: http://www.pc-freak.net/blog/how-to-find-and-kill-abusers-on-openvz-linux-hosted-virtual-machines-few-bash-scripts-to-protect-openvz-centos-server-from-script-kiddies-and-easify-the-daily-admin-job/ which was a good basis for automatically killing processes, but still not ideal, since all it takes a file name change...
My ideal method for dealing with this would be to either drop the route for the offending container, or to suspend the container.
Anyone had to deal with this issue, or otherwise have suggestions?
Comments
Even if I don't have to deal with such attacks because I'm not a provider, I would suspend the container.
No, they're attacking other systems on the internet, but it's saturating our connection to the point where we can't get in to stop it. So Adam has to drive over to the datacenter and log in with an attached keyboard/monitor. He's closer than I am to it, but he's still 40 minutes away.
We've tried buying an alternate connection from our DC, through the private lan connecting all the nodes, so that we could get in, but they don't seem to be interested.
Yes
I was actually considering port limiting, but I was hoping there was a more robust solution. I'd rather not limit anything network related, if possible.
We're running Virtualizor.
You can achieve close to the same thing by terminating the offending clients, and taking steps to ensure they don't receive further services from you.
@Aldryic: We're definitely doing so. Most (all?) of the issues seem to be stemming from containers with poor passwords getting compromised. I feel somewhat bad that we have to terminate accounts that cause issues like this, but then again, this is 2012; passwords like "password" and "redhat" aren't really viable anymore.
Your clients are responsible for their own security. If their accounts are compromised and use for illegitimate activity you have a right to protect your network/customers. Obviously some advice on security in the first instance may be the better way to handle this but if it is a recurring problem that is damaging to your business, you need to do what is needed
IMHO, you can suspend, send a mail to the customer about the situation and give it an advice on how to solve the problem.
I'm assuming that some lovely person is running a password guesser at your login prompts. Is there a method of restricting such attempts to say 5 and then locking out the IP address making those attempts and sending you an email?
Dedicate a NIC for management to your host if you haven't already. This way you can maitain access to your machine during an attack.
Then shutdown the machine which is abusing your network
Suspend, why would you other clients suffer because someone can't or doesn't want to secure their password?
Most compromises are not due to guessing a password, but from an exploited service. Typically something like a nulled copy of IPB.
Does the VPS in question have abnormally high loads during the attack? We've had some clients who have had their VPSs compromised and run DOS attacks which we noticed by the amount of processes running in vzlist but the loads were low and impact to our nodes were minimal but it sounds like your attacks are causing significantly more damage than ours.
If that's the case and the loads are abnormally high, you should be able to run a script on the server to detect high loads and stop the VPS with the highest load (we have a script setup to monitor loads and generate reports, but it should be easy enough to include an action in the script of course it may target the incorrect VPS).
As I finished typing that I thought of something else. Have a script monitor the server load or connections and if either are to high limit the connections or block an IP. We have a manual script for this (<3 iptables) but it would be fairly easy to automate with some netstat magic.
Do you know what kind of denial of service attacks your clients are making? Is this a simple udp flood, or an outgoing tcp/ip flood? If it's an outgoing udp flood, you can watch the dgramrcvbuf value under /proc/user_beancounters. If your customer isn't sending any outgoing udp packets, then the value will be 0. If it's performing a udp flood, you'll probably see something around 30400.
If it's UDP floods, limit just UDP to 10mbps.
Thanks for the input guys!
@kujoe: That's what I was thinking. I'm not sure that I can/should do it by load limit, since the vps plans that are having these issues are all limited to a single CPU core. So it'll have to be connections.
@ub3r: Thanks for the lead! I'm thinking that'll be a good starting point.
@dmmcintyre3: Good idea here too. I know how to limit packets with iptables, I'm thinking that bandwidth will be the realm of traffic shaping. I'll see if I can get the other solutions to be viable first.
@Drmike Fail2Ban can do that!
@Daniel Indeed fail2ban can do that.. I'm not sure if it's robust enough to be able to pick up SSH attempts on VPS containers when it's ran from the host nodes. But I haven't looked into it yet either, so maybe it can?
I don't think so.
Fail2ban reads the syslog file (of the host in this case), but can't read the syslog file in the containers (unless you havey tweak it, I think).
iptables is the way to go.
@Damian4IPXCore
Hmm, only way I can think of doing that is to check each containers auth.log
You could of course just put fail2ban in the templates, but it takes up a few mb of RAM
Could work, but will breaked if your customer change the ssh server from sshd to dropbear, for example.
Fail2ban read log file with qualified paths (/var/log/auth.log, for example). If you can read, from the host, the container hd you can use fail2ban. But you must set a rule for each container.
Still will be breaked when the customer change the sshd server.
@Steve81
Well, I don't think it interacts with the OpenSSH server, it just looks at the auth log
True, but different ssh server will write different things in different format in the log.
Fail2ban use, with openssh, the sshd filter. With dropber the dropbear filter.
Also Dropbear require a patch to write enough information in auth.log (without the path it don't write the client ip).
Sounds obvious to suspend the client. Be like "hey man you can't do that here what's wrong with you?"
We do suspend the client, we just can't get "there" to do so until our connection isn't saturated.
We finally made progress with the DC; they're going to provide us an alternate, 1mbit/sec connection for $30 per month. Seems pricey, but since all of us get alerts on all of our devices that something is down, we'll be able to log in and kill the offending container within a few seconds instead of the 40+ minutes it takes to drive to the DC.
Thanks for your input, everyone!
If your servers are on 100Mbit, limit your clients to 90Mbit,
If your servers are on 1Gbit, limit them to 900Mbit,
They will never notice, but it means they can't as easily fill up the pipe.
As soon as you read content from the users container (auth.log et al) be sure to mention that in the offer for the VPS (before the customers buys it) because this is otherwise called a serious breach of contract privacy-wise.
That's something that I would hope every host has... $30 a month doesn't seem that unreasonable to me.
Maybe it just seems expensive because we're paying 1/6th that per-megabit for our "normal" connection. Although I'm sure we've lost more than $30 this month due to downtime and lost business, so it'll definitely be worth it.
$30 a month does seem high until you compare it to what it would cost you to drive to the datacenter.
I know we've never looked into them but they do make remote KVM switches. At least I think that's what they're called. Maybe someone with more experience with them can chip in here....