Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


What to do after server has been compromised?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

What to do after server has been compromised?

valkvalk Member
edited April 2021 in Help

Greetings,

So 2 days ago to be exact, a bad actor just came to my server and doing malicious nasty stuff, and I got an abuse report about my server is being used to SSH bruteforce just yesterday when I was away from my computer, doing stuff at work, by the time I didn't notice anything until I noticed my image crawler was down, it was too late, so what things should I do now after that incident, currently looking to get rid the IP from blacklists

Don't hurt me about I didnt securing server shit, I'm tired of it already

Comments

  • Reinstall and next time use ssh keys instead of passwords.

  • pikepike Veteran

    Get a full backup of your compromised server, then wipe and restore from last uncompromised backup. Restore files that are missing from the compromised backup (carefully). Secure your server properly.

    Thanked by 2valk kkrajk
  • valkvalk Member

    @thedp said:
    Create a thread on LET.

    Eh my mind was really messed up dont ask

  • FalzoFalzo Member

    do not just blindly reinstall.

    stop all services running and try to find log-entries that match the timestamp from the abuse report. also you might want to search for files that have been modified or created around that time and seem to not belong there. check cron entries and whatnot.

    also 'abuse about ssh bruteforce' is not clear enough... does that mean your server has been bruteforced or rather your server has been abused to try and bruteforce something else?

    you want to find the cause for whatever happened and ideally also the attacking vector which has been used to run malicious stuff on your system. just reinstalling bears a big risk of you missing that vector and soon being back here because you've been hacked again.

    once you're done with this you still can reinstall and secure against the real problem...

    Thanked by 1valk
  • deankdeank Member, Troll
    1. Hastily come up with an excuse for your customers. Blame your girlfriend if you must. Blame the world if you want. Anything but blaming yourself works.

    2. Find a backup that is probably safe. Follow your gut feeling. No time to look for what has caused the hack.

    3. Play some games to remove stress from your mind and body.

    4. Lazily look into what has caused the compromise. It's not important though. After all, it wasn't your fault, probably. It's the world man, the rotten world.

    Rinse and repeat if you are hacked again.

    Thanked by 4valk pbx skorupion Tony40
  • uninstall ssh

  • valkvalk Member

    @notarobo said:
    uninstall ssh

    No, that wont happen unless I have to

    1. Change SSH port
    2. No more password login. Switch to key based login.
    3. Disable root login. Use sudo
    4. Install a firewall
    5. Update system regularly and apply patches on time

    These are bare minimum.

  • shutdown -h now
    Enter rescue system , backup , rsync to remote server,
    Reinstall
    rersync and you are good to go !

  • FalzoFalzo Member

    @valk said: my server is being used to SSH bruteforce

    thanks for editing/correcting that to be more clear.

    if your server was part of an outgoing attack then most likely following all the advise of securing your own sshd install won't cut it.
    there is a rather high probability that the attacker came into your system on a totally different way...

    don't get me worng, of course securing your sshd always makes sense.
    but as said above: you want to know the real cause/entry point to be able to prevent it from happening again.

  • JarryJarry Member

    @momkin said:
    shutdown -h now

    I'm not sure if it is good idea to trust anything on compromised system. You can never know what the attacker hooked to "shutdown". Maybe it will wipe out the whole disk instead to cover the traces?

    Thanked by 1yoursunny
  • valkvalk Member

    @Jarry said:

    @momkin said:
    shutdown -h now

    I'm not sure if it is good idea to trust anything on compromised system. You can never know what the attacker hooked to "shutdown". Maybe it will wipe out the whole disk instead to cover the traces?

    The server is already shut off for safety

  • JarryJarry Member

    Then there's no need to do "shutdown -h now", right?

    Thanked by 1valk
  • Let us know if you manage to find how the attacker got in.

  • jarjar Patron Provider, Top Host, Veteran

    Top answer here is the single best ever I’ve ever read: https://security.stackexchange.com/questions/39231/how-do-i-deal-with-a-compromised-server

    Thanked by 2Boogeyman freednsdk
  • @valk said:
    Greetings,

    So 2 days ago to be exact, a bad actor just came to my server and doing malicious nasty stuff, and I got an abuse report about my server is being used to SSH bruteforce just yesterday when I was away from my computer, doing stuff at work, by the time I didn't notice anything until I noticed my image crawler was down, it was too late, so what things should I do now after that incident, currently looking to get rid the IP from blacklists

    Don't hurt me about I didnt securing server shit, I'm tired of it already

    Backup your data and perform a clean re-install.

  • Have you heard of periods?

  • jarjar Patron Provider, Top Host, Veteran

    @TimboJones said:
    Have you heard of periods?

    Why, do you need a pad? I might have an extra.

    Thanked by 1NobodyInteresting
  • reinstall, use keys , disable login with passwords.

  • jvnadrjvnadr Member
    edited April 2021

    You have to give some more info about what's in your server. Is it for shared hosting? Is it contains a CMS like wordpress? What is the usage of?

    If you have access to the compromised server, bring it back online and disable all external access except from your own ip, to investigate the hacking. Then do the following:
    => Change keys, disable password login.
    => Check all the logs, timestamps, scan with antivirus and rootkits like Chkrootkit, Lynis, LMD etc.
    => If a CMS is isntalled, check for compromised / unstable plugins
    => Use tools like suIP.biz or SQL Injection Test for SQL injection vulnerabilities.

    When done with that, you will probably have a picture on what caused the hacking.

    Make sure that the computer that is used to enter to this server isn't compromised also (e.g. hacked laptop that helped the hacker to get your ssh password/ssh key).

    From here, I would suggest not to fix holes but to clean and install from fresh the server and reinstall only what is needed and only if it has been checked.

    So, generally speaking, most of the answers here are on the one way direction: backup the needed stuff (if it's a CMS like wordpress, for example, backup site/sites and reinstall). If you can recreate services without backing up (e.g. a vpn, a monitoring server etc.), then, just reinstall.

    Reinstall the server and do basic security actions:

    • Create key pair and disable root password login
    • Create a sudo user for safety with a really strong username and password
    • You can change port 22 for ssh, though it is not really critical
    • Update and full upgrade your system
    • Install fail2ban
    • Use SELinux if possible.
    • Install a firewall like UFW or/and CSF
    • Configure CSF to prevent brute force attacks
    • Change frequently the user's passwords
    • Monitor suspicious log messages with Logwatch
    • secure any web server you install (apache or nginx). There are plenty of tutorials on the web
    • Completely disable ftp, if present. Use only sftp with jailed users.

    Those steps are more than enough, for the maximum security. And the most important step at all: always keep offsite and local (downloaded) backups. The more valuable data, the more backups you need.

    Thanked by 1valk
  • jvnadrjvnadr Member
    edited April 2021

    @valk said: what things should I do now after that incident, currently looking to get rid the IP from blacklists

    Blacklists are useful to be cleaned only if the server is used for mail. So, you have to manually delist the ip from each of the services is listed on. Each one has it own procedure, you need to search it.
    But usually, a hacked server is not lead to blacklisted ip - at least, not automatically. What lists are the ones that blacklisted your ip?

    If you have been blocked from services like google (red alert for malicious site, etc.), then, it isn't the ip that is affected, but the url. So, you need to follow procedure that each blocking service uses (like Google) to delist it, explaining the measures you took to clean the server and secure it not to have incidents again.
    It is often a painful job, but it has to be done if you want to keep ip and url...

  • darbdarb Member

    @valk said:
    Greetings,

    ... so what things should I do now after that incident, ...

    It would be helpful to know:

    1.) what you did to harden your server before.
    2.) what software you were/are running.
    3.) how often you were doing updates.

    Thanked by 1Tony40
  • Tony40Tony40 Member
    edited April 2021

    The first thing I do when I get a Server or VPS box.
    1- apt update & apt upgrade
    2- change SSH port.
    3- Add a user.
    4- Disable root login.
    5- in etc/ssh/sshd_config AllowUsers [email protected] SSH login by IP only.
    6- Set-up Fail2Ban for intrusion prevention and brute-force attacks.
    7- Set-up SSH successful login email alert notification.

    Thanked by 1valk
  • valkvalk Member

    @darb said:

    @valk said:
    Greetings,

    ... so what things should I do now after that incident, ...

    It would be helpful to know:

    1.) what you did to harden your server before.
    2.) what software you were/are running.
    3.) how often you were doing updates.

    I am running Debian Bullseye b4 the incident, and I just left with the default config as the installation it goes, doing daily updates. It's my bad for leaving the server unsecured

  • @valk said:

    I am running Debian Bullseye b4 the incident, and I just left with the default config as the installation it goes, doing daily updates. It's my bad for leaving the server unsecured

    Just re-install the OS. If the server was already compromised, I would not trust anything in it!

  • ploxhostploxhost Member, Patron Provider

    Wipe the machine. Not worth taking the risk if you were compromised ;)

  • wilbowilbo Member

    Trash machine and get a fresh new one. The old one is forever unclean!!

  • @valk said:

    @darb said:

    @valk said:
    Greetings,

    ... so what things should I do now after that incident, ...

    It would be helpful to know:

    1.) what you did to harden your server before.
    2.) what software you were/are running.
    3.) how often you were doing updates.

    I am running Debian Bullseye b4 the incident, and I just left with the default config as the installation it goes, doing daily updates. It's my bad for leaving the server unsecured

    well ur running a test build live then

Sign In or Register to comment.