Something is terribly wrong with my VM

xaitmi

Hi so I have a VM at DigitalOcean, Centos 6 64 bit. 512 mb ram, etc

I run CentMinMod and it uses fail2ban and csf firewall.

Earlier today I guess someone decided to DDOS my site using Layer7 or something, and everything went haywire.

Here are some pics

As you can see a few hours ago a spike, and then my VM froze, then it kernel panicked.

Digital Ocean support said my console is being flooded with the firewall blocked messages as you can see in the pic above, I tried rebooting the vps and went to console and its still happening.

I can't even backup my files or even use the VM because the firewall messages wont stop which probably means the attackers are still attacking it from multiple ips causing the firewall to go nuts.

What should I do now?

Note: I do use cloudflare on my website with "I'm under ddos mode" on at all times, but with the recent Cloudflare CNAME exploit anyone can attack anyones site no matter what plan their on, as long as their using CF.

joepie91

xaitmi said: Note: I do use cloudflare on my website with "I'm under ddos mode" on at all times, but with the recent Cloudflare CNAME exploit anyone can attack anyones site no matter what plan their on, as long as their using CF.

Oh, another one? What's the bypass this time?

nightshade

Would something like this work.

1. Using the console shutdown eth0. Maybe DO can do this if you can't even get working access from console.

2. Snapshot image

3. Create new droplet from snapshot. Do what you need to do with new image. It will be on new ip and shouldn't be under attack.

4. Destroy old droplet.

xaitmi

@joepie91 said:
Oh, another one? What's the bypass this time?

"Cloudflare has an exploit that's being used to bypass their security. It's related basically to CNAME DNS entries. Basically anyone can setup a cname pointing to HF then attack that cname entry and it will 100% bypass the Cloudflare protection scheme. "

If you go on google and type in Cloudflare CNAME exploit you will find a hacking forum on the 3rd or 4th result where the admin talks about it, and how it's causing his site to be ddosed.

xaitmi

@nightshade said:
Would something like this work.

1. Using the console shutdown eth0. Maybe DO can do this if you can't even get working access from console.

2. Snapshot image

3. Create new droplet from snapshot. Do what you need to do with new image. It will be on new ip and shouldn't be under attack.

4. Destroy old droplet.

Thanks taking a snapshot now and replying to the ticket

Fidde

During snapshots the droplets can't be powered on so if you go that route you should have no problems as long as they don't start attacking your new IP.

joepie91

xaitmi said: Basically anyone can setup a cname pointing to HF then attack that cname entry

Does this refer to setting up such a record within Cloudflare itself? Because other than a bug in the CNAME flattening they apply, I can't see how this could possibly work.

xaitmi

Thanks guys, I got my site back up and running now using the Snapshot method.

Also DO replied to my ticket with exactly the same steps @nightshade said in his post above :P

Setting up a nginx reverse proxy now on a MyCustomHosting spare VPS to use on my site.

Going to use the following setup

CF --> NGINX Reverse Proxy --> DigitalOcean

This way if someone does use the CNAME exploit to attack my site, the reverse proxy will take the hit.

xaitmi

@joepie91 said:
Does this refer to setting up such a record within Cloudflare itself? Because other than a bug in the CNAME flattening they apply, I can't see how this could possibly work.

http://webcache.googleusercontent.com/search?q=cache:zm3hF4RpRLgJ:www.hackforums.net/archive/index.php/thread-4811061.html+&cd=8&hl=en&ct=clnk&gl=ca

joepie91

xaitmi said: http://webcache.googleusercontent.com/search?q=cache:zm3hF4RpRLgJ:www.hackforums.net/archive/index.php/thread-4811061.html+&cd=8&hl=en&ct=clnk&gl=ca

Yeah, that's what I was reading. It doesn't clarify on how the attack/bug actually works, though.

jar

@joepie91 said:
Yeah, that's what I was reading. It doesn't clarify on how the attack/bug actually works, though.

Yeah it simply doesn't make sense to me at all. I think people really over assume what a CNAME record is. Attacking one is literally adding a DNS lookup to say "attack the A record of this other hostname."

That said...maybe CloudFlare's DDOS protection is reliant on the hostname being requested and they're simply passing the traffic as though the protection is disabled via the panel? But then wouldn't that also assume dedicated IPs and that attacking the CF IP directly would be an identical attack?

I don't think this is a thing, but maybe I'm wrong.

PrincessOfCats

@Jar said:
I don't think this is a thing, but maybe I'm wrong.

I know that for cPanel, if you use the Cloudflare plugin, Cloudflare requires you to have a A record pointing to the server, though that doesn't explain what's going on at the moment.

@xaitmi

You can still blindly login, the console does not prevent you from doing that.

Do this:

1. Reboot and wait for a while to make sure that the console is up
2. Blindly type your username, hit enter
3. Blindly type your password, hit enter
4. csf -x, messages should now stop
5. Setup iptables to reject all except for established sessions.
6. Backup

rds100

It's is a very bad idea to setup your firewall to log every dropped packet. This makes it very easy to kill the box with some moderate pps flood.

PrincessOfCats

@rds100 said:
It's is a very bad idea to setup your firewall to log every dropped packet. This makes it very easy to kill the box with some moderate pps flood.

Depending on setup, it is possible that there is a combination of LFD writing new blocked ips rapidly to /etc/csf/csf.deny + the CSF logs in /var/log/syslog.

That being said, you can adjust those settings in /etc/csf/csf.conf.

###############################################################################
# SECTION:Logging Settings
###############################################################################
# Log lfd messages to SYSLOG in addition to /var/log/lfd.log. You must have the
# perl module Sys::Syslog installed to use this feature
SYSLOG = "0"

# Drop target for iptables rules. This can be set to either DROP ot REJECT.
# REJECT will send back an error packet, DROP will not respond at all. REJECT
# is more polite, however it does provide extra information to a hacker and
# lets them know that a firewall is blocking their attempts. DROP hangs their
# connection, thereby frustrating attempts to port scan the server.
DROP = "DROP"

# Enable logging of dropped connections to blocked ports to syslog, usually
# /var/log/messages. This option needs to be enabled to use Port Scan Tracking
DROP_LOGGING = "1"

# Enable logging of dropped incoming connections from blocked IP addresses
#
# This option will be disabled if you enable Port Scan Tracking (PS_INTERVAL)
DROP_IP_LOGGING = "0"

# Enable logging of dropped outgoing connections
#
# Note: Only outgoing SYN packets for TCP connections are logged, other
# protocols log all packets
# 
# We recommend that you enable this option
DROP_OUT_LOGGING = "1"

# Only log incoming reserved port dropped connections (0:1023). This can reduce
# the amount of log noise from dropped connections, but will affect options
# such as Port Scan Tracking (PS_INTERVAL)
DROP_ONLYRES = "0"

# Commonly blocked ports that you do not want logging as they tend to just fill
# up the log file. These ports are specifically blocked (applied to TCP and UDP
# protocols) for incoming connections
DROP_NOLOG = "67,68,111,113,135:139,445,500,513,520"