VPS suspended

JohnRoe

can anyone help me analyse this log?
my vps got suspended and the provider ask what am I hosting..
I just host a small website with regular mysql usage..and my website are up less than a few days..
I dont know what this log saying.. I censored my server ip

http://pastebin.com/TeuRxDsM

SandwichBagGhost

Looks like your VPS was sending out DoS or you got attacked by a DoS attack.

vonlulzweg

your webserver got attacked

http://stackoverflow.com/questions/11729517/how-to-stop-syn-sent

zevus

so this is just a small portion of the log, right?

all those [unreplied] means, well, server isnt (or wont) acknowledge those connections.

why do you keep trying to open 162.218.30.18:80

JohnRoe

my server sending DoS or it got attacked by DoS??

@zevus yup.. my provider gave it to me.. I dont recognize that ip.. my website will only process some row of mysql when a user use my website

zevus

Well, something is making it open connections to 162.218.30.18:80 over and over. I'm guessing to be suspended that if we had timestamps it'd probably be at least once a second if not multiple times a second..

NetRange: 162.218.30.16 - 162.218.30.23

CustName: Shen Bifen

Address: tiyuchangdongmenailouxinghunshseying

City: hangzhoushi

StateProv: ZEJIANGSHENG

PostalCode: 310000

Country: CN

RegDate: 2013-12-17

Updated: 2013-12-17

JohnRoe

so, if my provider boot up the server, will it send the request automatically? I dont want it to be suspended twice T_T

LES

Reinstall!

JohnRoe

@zevus said:
Well, something is making it open connections to 162.218.30.18:80 over and over. I'm guessing to be suspended that if we had timestamps it'd probably be at least once a second if not multiple times a second..

NetRange: 162.218.30.16 - 162.218.30.23

CustName: Shen Bifen

Address: tiyuchangdongmenailouxinghunshseying

City: hangzhoushi

StateProv: ZEJIANGSHENG

PostalCode: 310000

Country: CN

RegDate: 2013-12-17

Updated: 2013-12-17

china??? damnn... my other server got attacked before this.. they bruteforce my server..and they managed to get into it.. after that they use my server to send weird request.. and the cpu load was very heavy.. CHINA -_-

JohnRoe

@LES said:
Reinstall!

I always reinstall when something happened on my server.. because I scared fixing only last for a short time

SandwichBagGhost

1) Disable password authentication for SSH
2) Use SSH public key authentication for SSH only
3) Optional; disable root login via SSH (I don't see why you'd want to do this if you have SSH public key auth, when no one ever gets your private key and its passphrase everything should be fine with it being enabled)

JohnRoe

@SandwichBagGhost said:
1) Disable password authentication for SSH
2) Use SSH public key authentication for SSH only
3) Optional; disable root login via SSH (I don't see why you'd want to do this if you have SSH public key auth, when no one ever gets your private key and its passphrase everything should be fine with it being enabled)

I never use key for auth.. maybe it is the time for me to learn that

SandwichBagGhost

You should use SSH public key authentication with a passphrased key instead of password authentication.

Of course a passphrase less key allows you to do password less login but it's less secure. If someone steals your key (shouldn't happen when you are careful) he can login into your server if the key has no passphrase but if the key has one he first has to get the passphrase to be able to login.

Bruteforce on SSH key authentication systems is impossible as you get kicked out instantly because of a not supported authentication type (as you disabled password authentication before).

JohnRoe

@SandwichBagGhost said:
You should use SSH public key authentication with a passphrased key instead of password authentication.

Of course a passphrase less key allows you to do password less login but it's less secure. If someone steals your key (shouldn't happen when you are careful) he can login into your server if the key has no passphrase but if the key has one he first has to get the passphrase to be able to login.

Bruteforce on SSH key authentication systems is impossible as you get kicked out instantly because of a not supported authentication type (as you disabled password authentication before).

ok then.. thanks for that advice.. I think I will use in on all my server.. thanks again

SandwichBagGhost

The above said does of course not protect you from security holes in other programs that may allow hackers to run any code they want as root.

Always keep your system up to date and secured.

vonlulzweg

Your vps has not been hacked but got attacked by SYN FLOOD. See SYN_SENT in your log.

http://stackoverflow.com/questions/11729517/how-to-stop-syn-sent

Droidzone

SandwichBagGhost said: Disable password authentication for SSH

Or have something like: O8&0T@KyUr9yPSuVamxij8esdkPqCu1D&0IVmsj&lhsuG6ElecF8ZWnYMvL& ?

Gallaeaho

If the attacker successfully attacks and exploits known vulnerabilities in the software you run on your VPS, then no amount of extra long passwords or key authentication is going to save you from the inevitable good-server-gone-bad deal. As a server owner, you need to make sure that you're keeping your software updated regularly.

I'm sorry for essentially copying @SandwichBagGhost's post, but I will give credit where it is due. They're right on the money.

JohnRoe

Yes my vps is uptodate because I bought that server on 25/8 and install webserer and run my website on 26/8 since then, software are uptodate..
but, how the attacker found my ip or domain?? I bought the domain 3 days ago..

aggressivenetworks

Did you change your ssh ports? Change your password from something capitals letters, complex and symbols. No passwd like this pass23456. You install fail2ban? Use Cloudfare or use Nginx as a reverse proxy to forward traffic to web server. Rate limit connections with iptables. Optimize your web server configuration. Make sure you a audit your logs at least weekly.

linuxthefish

Looks like SYN flood in, default nodewatch settings pick this up and suspend. I had this issue with GVH yesterday (not their fault), when I had a massive flood of 'traffic'.

Cloudflare helped in my situation, so you could try that if you are running a website.

emg

@LES said:
Reinstall!

@psycholyzern said:
I always reinstall when something happened on my server.. because I scared fixing only last for a short time

Yes, but what will you change from the last time you reinstalled to prevent a recurrence?

JohnRoe

@aggressivenetworks said:
Did you change your ssh ports? Change your password from something capitals letters, complex and symbols. No passwd like this pass23456. You install fail2ban? Use Cloudfare or use Nginx as a reverse proxy to forward traffic to web server. Rate limit connections with iptables. Optimize your web server configuration. Make sure you a audit your logs at least weekly.

unfortunately, I still use port 22 and use dictionary password.. this is because I just bought the server and the domain.. I dont know people will found it too quick..

@emg said:
Yes, but what will you change from the last time you reinstalled to prevent a recurrence?

reinstall and apply what i need to apply to prevent that problem from happening again

ahmiq

from stackoverflow:

This question seems to be getting many views but yet no answer, so I decided to answer my own question for anyone looking for a solution.

First thing first, knowing the reason is half of the solution. I was under what is called SYN Flooding Attack which uses HTTP protocol behavior against itself

The short of it is, remote client tries to establish a connection with your server by sending SYN, your server replies with SYN_ACK (in your logs you will see SYN_SENT) and will wait until it receives ACK. If ACK is not received within xx seconds, your server will send SYN_ACK again, .... and again.... and again. It will eventually reach the configured threshold and stop accepting any more SYN request making your server unresponsive. One of the symptoms which happened to me was that my website was responding once like nothing is wrong but not responding in the next xx times.

The solution that worked for me was enabling SYN cookies, SSH into your server, open the following file using your favorite editor. I'm using vi in this example

vi /etc/sysctl.conf
And add these lines to the file, then restart your server. Hopefully this will stop the attack as it did for me

net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 3
I was using CentOS, I think the above solution will work on all distributions but in case it didn't search for "How to stop SYN Flooding Attack" for your linux distribution

On a side note, blocking the IPs initiating the SYN requests will probably not help because most likely the attacker has spoofed the IPs

JohnRoe

T_T my provider unsuspended my server few times.. but my server keep going suspended again after few secs/mins being online T_T
btw I applied stackoverflow.com/questions/11729517/how-to-stop-syn-sent but still my vps being suspended again..
I didnt have backup of my ssl key and they only issue ssl once.. so, I would lost ssl if I reinstall T_T

JustAMacUser

Your server is compromised, you can't (shouldn't) be using that SSL key anyway. You should request it be revoked and re-issue the cert with a new key... After you reinstall your system.

Take the advice given to you already: reinstall the system and use key-based authentication for SSH. If possible, restrict SSH connections to a single IP (e.g. your home/office IP if it's static).