New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
What to do after server has been compromised?
Greetings,
So 2 days ago to be exact, a bad actor just came to my server and doing malicious nasty stuff, and I got an abuse report about my server is being used to SSH bruteforce just yesterday when I was away from my computer, doing stuff at work, by the time I didn't notice anything until I noticed my image crawler was down, it was too late, so what things should I do now after that incident, currently looking to get rid the IP from blacklists
Don't hurt me about I didnt securing server shit, I'm tired of it already
Comments
Reinstall and next time use ssh keys instead of passwords.
Create a thread on LET.
Get a full backup of your compromised server, then wipe and restore from last uncompromised backup. Restore files that are missing from the compromised backup (carefully). Secure your server properly.
Eh my mind was really messed up dont ask
do not just blindly reinstall.
stop all services running and try to find log-entries that match the timestamp from the abuse report. also you might want to search for files that have been modified or created around that time and seem to not belong there. check cron entries and whatnot.
also 'abuse about ssh bruteforce' is not clear enough... does that mean your server has been bruteforced or rather your server has been abused to try and bruteforce something else?
you want to find the cause for whatever happened and ideally also the attacking vector which has been used to run malicious stuff on your system. just reinstalling bears a big risk of you missing that vector and soon being back here because you've been hacked again.
once you're done with this you still can reinstall and secure against the real problem...
Hastily come up with an excuse for your customers. Blame your girlfriend if you must. Blame the world if you want. Anything but blaming yourself works.
Find a backup that is probably safe. Follow your gut feeling. No time to look for what has caused the hack.
Play some games to remove stress from your mind and body.
Lazily look into what has caused the compromise. It's not important though. After all, it wasn't your fault, probably. It's the world man, the rotten world.
Rinse and repeat if you are hacked again.
uninstall ssh
No, that wont happen unless I have to
These are bare minimum.
shutdown -h now
Enter rescue system , backup , rsync to remote server,
Reinstall
rersync and you are good to go !
thanks for editing/correcting that to be more clear.
if your server was part of an outgoing attack then most likely following all the advise of securing your own sshd install won't cut it.
there is a rather high probability that the attacker came into your system on a totally different way...
don't get me worng, of course securing your sshd always makes sense.
but as said above: you want to know the real cause/entry point to be able to prevent it from happening again.
I'm not sure if it is good idea to trust anything on compromised system. You can never know what the attacker hooked to "shutdown". Maybe it will wipe out the whole disk instead to cover the traces?
The server is already shut off for safety
Then there's no need to do "shutdown -h now", right?
Let us know if you manage to find how the attacker got in.
Top answer here is the single best ever I’ve ever read: https://security.stackexchange.com/questions/39231/how-do-i-deal-with-a-compromised-server
Backup your data and perform a clean re-install.
Have you heard of periods?
Why, do you need a pad? I might have an extra.
reinstall, use keys , disable login with passwords.
You have to give some more info about what's in your server. Is it for shared hosting? Is it contains a CMS like wordpress? What is the usage of?
If you have access to the compromised server, bring it back online and disable all external access except from your own ip, to investigate the hacking. Then do the following:
=> Change keys, disable password login.
=> Check all the logs, timestamps, scan with antivirus and rootkits like Chkrootkit, Lynis, LMD etc.
=> If a CMS is isntalled, check for compromised / unstable plugins
=> Use tools like suIP.biz or SQL Injection Test for SQL injection vulnerabilities.
When done with that, you will probably have a picture on what caused the hacking.
Make sure that the computer that is used to enter to this server isn't compromised also (e.g. hacked laptop that helped the hacker to get your ssh password/ssh key).
From here, I would suggest not to fix holes but to clean and install from fresh the server and reinstall only what is needed and only if it has been checked.
So, generally speaking, most of the answers here are on the one way direction: backup the needed stuff (if it's a CMS like wordpress, for example, backup site/sites and reinstall). If you can recreate services without backing up (e.g. a vpn, a monitoring server etc.), then, just reinstall.
Reinstall the server and do basic security actions:
Those steps are more than enough, for the maximum security. And the most important step at all: always keep offsite and local (downloaded) backups. The more valuable data, the more backups you need.
Blacklists are useful to be cleaned only if the server is used for mail. So, you have to manually delist the ip from each of the services is listed on. Each one has it own procedure, you need to search it.
But usually, a hacked server is not lead to blacklisted ip - at least, not automatically. What lists are the ones that blacklisted your ip?
If you have been blocked from services like google (red alert for malicious site, etc.), then, it isn't the ip that is affected, but the url. So, you need to follow procedure that each blocking service uses (like Google) to delist it, explaining the measures you took to clean the server and secure it not to have incidents again.
It is often a painful job, but it has to be done if you want to keep ip and url...
It would be helpful to know:
1.) what you did to harden your server before.
2.) what software you were/are running.
3.) how often you were doing updates.
The first thing I do when I get a Server or VPS box.
1- apt update & apt upgrade
2- change SSH port.
3- Add a user.
4- Disable root login.
5- in etc/ssh/sshd_config AllowUsers [email protected] SSH login by IP only.
6- Set-up Fail2Ban for intrusion prevention and brute-force attacks.
7- Set-up SSH successful login email alert notification.
I am running Debian Bullseye b4 the incident, and I just left with the default config as the installation it goes, doing daily updates. It's my bad for leaving the server unsecured
Just re-install the OS. If the server was already compromised, I would not trust anything in it!
Wipe the machine. Not worth taking the risk if you were compromised
Trash machine and get a fresh new one. The old one is forever unclean!!
well ur running a test build live then