New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
My VPS has been compromised!
Today I tried to change something on my VPS and I noticed it was slow as hell. I saw a lot of apache2 processes so I started to investigate.. I found strange requests in my apache2 log: http://pastebin.com/vUg4hskZ
So I googled my IP... I was not happy with the results: http://bestblackhatforum.com/Thread-546-L1-L2-Proxies-15-Apr-2013 (My IP is in there).
How can I investigate how they got in? Apparantly my VPS has been compromised since the 27th of March: http://www.stopforumspam.com/ipcheck/37.247.51.11o (replace last character with a 0)
Comments
If they weren't thorough, you might be lucky enough to find something in your auth/secure log.
run netstat -antup
there may be still current malicious outbound connections.. they often use these as irc/spam bots etc once they get in..
when you see a malicious looking connection, find the PID, then run:
lsof -p PID
this will often give you the path to the malicious script/binary.. (often in /tmp or /dev/shm)
Did you keep it updated ? Do you run wordpress ? Did you keep that updated as well as the plugins, did you keep all your scripts updated ?
It looks like an apache hack. Might even not got root access.
But, anyway, we are receiving abuse complaints every day, for rooted boxes it is much sooner as they scan various honeypots, but apache hacks are only used for proxy or hosting illegal content/malware and is harder to appear on reports.
To get a more readable overview of what processes have connections open (or are listening on ports):
lsof -iTo find logins that aren't you:
cat /var/log/auth.log | grep Acceptedorcat /var/log/secure | grep Accepted(depending on distro).whoandlastare also useful.Do a
cat /etc/passwdto see if any suspicious accounts have been added.Your package manager logs may also tell you more, depending on whether the intruder has root access or not.
EDIT: Almost forgot to mention... copy all your Apache logs and auth logs to your local machine right now before anyone has a chance to delete them, so that you can look through them thoroughly.
do you have any panel? or firewall protection?
ssh on port 22? fail2ban?
Firewall is hyped. Unless you need to block some ranges or rate-limit stuff, as long as your service listens to a port and the firewall has to allow it, that service will be compromised if has a vulnerability which is known.
The firewall can then be tweaked to allow access for the hacker's service presuming escalation was successful and they have root access.
From OP description and our lack of reports so far, I think this is only apache hack and root escalation was not successful or not tried.
Thanks for all the quick replies, I really do appreciate it!!
Different distro, Ubuntu Server 12.04 but thanks for the pointer.
Is that a Linux command? I only seem to find Windows usage examples?
Thanks, investigating!
Thanks, Joepi91, that's a really helpful command! Appreciate it!!!
It looks like an apache hack. Might even not got root access.
I updated it the last time 2 weeks ago. I did not run Wordpress, this box was running a default installation of Ubuntu together with ZPanel.
It indeed looks like an apache hack only. Is it a known hack at the moment? How can I disable this proxy crap thingy? I don't want a proxy!
Looks clean!
No suspicious accounts!
Done! I cannot download the 'other_vhosts_access.log' log files, as in total they are about 10GB in filesize!! I seem to be abused pretty hard.
I was running ZPanel. No firewall in place. They were abusing port 80 by the looks of it.
SSH on different port. No fail2ban
If you don't need it, unload the mod_proxy module in httpd.conf, just toss a # at the beginning of the pertinent #LoadModule line...
Yep, you'll find netstat on almost any OS, tab it out man!
Update; Auth logs look clean, thankfully. Only suspicious I could find is this:
Mar 30 02:48:23 data apache2: gethostby.getanswer: asked for "rrlib.cs.uni-kl.de IN AAAA", got type "DNAME"
Mar 30 02:48:23 data apache2: gethostby.getanswer: asked for "rrlib.cs.uni-kl.de IN A", got type "DNAME"
Also, it seems that they were abusing 'mod_proxy'. Maybe it was mis configured, or it indeed has a security hole.
And after that, the sh!tstorm starts and my access log is FLOODED with requests.
If you say your server is compromised, then take all the important files off. Delete the VPS are rebuild it from scratch. End of story.
That's likely to be the problem. That said, I would recommend not running ZPanel if you don't have an absolute need for it.
Or if you host at Burst, they'll do it for you without notice
+1 @jhadley
Remember to get a backup of the access.log before deleting anything. The VPS was probably used to commit crimes and you need to have proof that they weren't commited by you.
And how does he prevent the same thing happening again? Re-installing from scratch without understanding the underlying issue isn't a solution.
I disabled mod_proxy but as soon I enable/start apache2, about 100-200 IPs try to connect to my VPS on port 80 and it shows established..
Done...
Agreed they were just, as it appears, using him for a proxy + ad serve That is almost intentionally obvious
You're on the books as a proxy server Brah, If you need to serve http requests then it's Time to change Ip
@Freek and BTW not that you couldn't do some thing, But if I had to keep the Ip I'd get creative ya know
The more appropriate thing I shoulda' said is What you gonna go about the free for all broadcast
If you can't force the Thread down a change in Ip is a foregone conclusion
Im not saying to not understand the underlying issue, what I am saying is don't put it back in a production setup. The hacker could have created an ssh key to get in. He could have made a mysql account to look at the databases. Who knows. The fact is that if this would have happened at a 'business' that had competent techs they would delete the server and start over.
Not sure how people get compromised, my password is secret. thisisnotmypass123456
Yeah I'm pretty much screwed now. I do indeed need to server HTTP and the fact that this frigging mod_proxy crap was running on port 80 as well doesn't make it any easier (else I could have iptables block all traffic on port XXX, done).
I'm off to bed now. Thanks for the quick assistance guys! Appreciate it, really do!!
No.
Go read about CSF/LFD. Multiple failed logins or other bad activity at the application level triggers iptables rule that blocks any further connections from that IP for a period of time. Very effective in shutting down port scanning, brute-forcing, etc.
you should install CSF firewall, there you can limit many apache stuff.
queries per second, etc, etc.
but of course that is before beeing compromised. <- forget that, INSTALL IT !!
A competent tech would archive the infected install and do forensics to to identify the root cause. "Deleting and starting over" isn't a solution to the issue.
True, but they won't throw it back into production. Which is my point. Don't put that particular instance back into the work flow as it will only end up being bad.
+1..and while there are always zero-days this screams mis configuration imho
First things I do when I get a VPS: Install fail2ban, disable root login, change the ssh port, get my public key in the server, and disable password auth.
Those are great first several steps to getting started, but this was mostly focused on issues with misconfiguration.