Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Advertise on LowEndTalk.com
Serious - Need help - CentOS8 - VM - Probably compromised
New on LowEndTalk? Please read our 'Community Rules' by clicking on it in the right menu!

Serious - Need help - CentOS8 - VM - Probably compromised

Ok, so finally it happened. A couple days ago, I installed centos8 on a VM... nothing else is configured. Its a brand new custom ISO install.

Today, I was checking the graphs/ stats and it looks like the server is maxing CPU for past 2 days and outbound network activity is happening at almost full port speed...

I am preplexed as to what may have happened. I have shut down the VM for now.. to make sure nothing more bad can happen (I think).

I know a quick fix is to reinstall... Since there is no data on the server as its a fresh install, I am OK to do that, but I would rather learn on troubleshooting this.

What is it something I can check to find out what was the reason for the outbound connectivity? Is my server compromised? If yes, what logs can I check? How can I know if some malicious user is wreaking havoc? What is it that was causing the 100% CPU (per providers chart)?

Any advise is appreciated.

Thanks in advance.

Comments

  • jackbjackb Member, Provider
    edited September 10

    Reboot into a rescue cd and inspect your logs for indicators of compromise.

    If I were a betting man $7 would say it's SSH bruteforce and whoever got in is launching DoS attacks. That would explain the traffic and the CPU.

    Thanked by 1skorous

    Afterburst - Awesome OpenVZ&KVM VPS in US+EU

  • Most likely compromised. I honestly would not even bother with logs since whether its compromised or not its not going to change the course of action which is just reinstalling.

    Make sure you harden your SSH after the reinstall.

  • @jackb said:
    Reboot into a rescue cd and inspect your logs for indicators of compromise.

    If I were a betting man $7 would say it's SSH bruteforce and whoever got in is launching DoS attacks. That would explain the traffic and the CPU.

    Would you be a bit specific on what logs may provide this info? Thnx

  • @stewdetox said:
    Most likely compromised. I honestly would not even bother with logs since whether its compromised or not its not going to change the course of action which is just reinstalling.

    Make sure you harden your SSH after the reinstall.

    Hmm. That seems to be the best option. But I was just trying to see if there is any rogue script somewhere going crazy...

    Interns of hardening, what are tour recommendations? Thanks

  • @plumberg said: Would you be a bit specific on what logs may provide this info?

    I'm not a CentOS person, but the equivalent of /var/log/auth.log should give you a clue on how/who logged in and from where (unless it's a more sophisticated attack which also wipes it's traces - which means it is going to be interesting to say the least). That should be your first goto place when you're inspecting the systems via rescue boot.

    @plumberg said: Interns of hardening, what are tour recommendations?

    Disable password logins entirely and go full fledged key based. Firewall your SSH port (white list your IPs if that is feasible). To reduce the typical junk script from scanning, try to change your SSH port to something very unusual (this is to reduce noise in your logs and not really "secure" your SSH).

    There are lots of other things - fail2ban for e.g. Pick your poison - there's enough well documented stuff on improving SSH security.

  • jackbjackb Member, Provider
    edited September 10

    @plumberg said:

    @jackb said:
    Reboot into a rescue cd and inspect your logs for indicators of compromise.

    If I were a betting man $7 would say it's SSH bruteforce and whoever got in is launching DoS attacks. That would explain the traffic and the CPU.

    Would you be a bit specific on what logs may provide this info? Thnx

    /var/log/secure

    To be clear I absolutely recommend reinstalling, but I think there's value in finding how your system was compromised. That way you know where you need to improve security measures for next time.

    Afterburst - Awesome OpenVZ&KVM VPS in US+EU

  • Looks like it got botrekt.

  • Have a look at top or htop to see the running processes. It will update in real-time and show you what’s using 100% CPU.

    It definitely sounds like a compromised system. Would always recommend securing SSH as a minimum after installing any OS.

  • Running RKhunter and the likes might help determine what it's infected with, I believe.

    Netstat to show open connections.

  • Since you're running a VM, I'd suggest you to re-install first because if this is either DDoS or SPAM or something shady like that, you're probably going to get your VM kicked off from their servers due to abuse.

    Re-install then follow a quality harden manual.

    I repeat, RAID is not backup | Looking for a developer for your next project? - Hire me

  • @sdglhm said:
    Since you're running a VM, I'd suggest you to re-install first because if this is either DDoS or SPAM or something shady like that, you're probably going to get your VM kicked off from their servers due to abuse.

    Re-install then follow a quality harden manual.

    If you're coming in via console just don't enable the network. Investigate as long as you want.

  • We have 5 honeypot systems configured to evaluate default security configuration in various CentOS, Redhat, Debian, Ubuntu, Mint versions. The only systems that were compromised were due to the following:
    1. SSH root password with less than 8 characters (no special characters)
    2. CentOS 7 base install without running yum updates
    3. Debian 6 base install vulnerabilities (with apt updates)
    4. Ubuntu 16 server base install without apt updates

    so based on our findings I assume that you either didn't run yum update after install or your SSH pass is less than 8 non-alphanumeric characters. What kind of customizations are done to your ISO?

    Thanked by 1raindog308
  • @jackb said:

    @plumberg said:

    @jackb said:
    Reboot into a rescue cd and inspect your logs for indicators of compromise.

    If I were a betting man $7 would say it's SSH bruteforce and whoever got in is launching DoS attacks. That would explain the traffic and the CPU.

    Would you be a bit specific on what logs may provide this info? Thnx

    /var/log/secure

    To be clear I absolutely recommend reinstalling, but I think there's value in finding how your system was compromised. That way you know where you need to improve security measures for next time.

    For some reason, there is no such file for me.

  • @TimboJones said:
    Running RKhunter and the likes might help determine what it's infected with, I believe.

    Netstat to show open connections.

    Thanks. I just installed it.

    [17:12:36] Warning: Checking for possible rootkit files and directories [ Warning ]
    [17:12:36] Found file '/etc/cron.hourly/gcc.sh'. Possible rootkit: BillGates botnet component
    [17:12:36]

    the file in cron:

    !/bin/sh

    PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/X11R6/bin
    for i in cat /proc/net/dev|grep :|awk -F: {'print $1'}; do ifconfig $i up& done
    cp /lib/libudev.so /lib/libudev.so.6
    /lib/libudev.so.6

    Not sure what is wrong here.... any suggestions?

  • @Hoptrics said:
    We have 5 honeypot systems configured to evaluate default security configuration in various CentOS, Redhat, Debian, Ubuntu, Mint versions. The only systems that were compromised were due to the following:
    1. SSH root password with less than 8 characters (no special characters)
    2. CentOS 7 base install without running yum updates
    3. Debian 6 base install vulnerabilities (with apt updates)
    4. Ubuntu 16 server base install without apt updates

    so based on our findings I assume that you either didn't run yum update after install or your SSH pass is less than 8 non-alphanumeric characters. What kind of customizations are done to your ISO?

    password is 8 char alpha numeric.
    Yes, I probably did not run yum update.
    Its a standard ISO provided by provider.. never had issue in general.

  • HoptricsHoptrics Member
    edited September 10

    @plumberg said:

    @Hoptrics said:
    We have 5 honeypot systems configured to evaluate default security configuration in various CentOS, Redhat, Debian, Ubuntu, Mint versions. The only systems that were compromised were due to the following:
    1. SSH root password with less than 8 characters (no special characters)
    2. CentOS 7 base install without running yum updates
    3. Debian 6 base install vulnerabilities (with apt updates)
    4. Ubuntu 16 server base install without apt updates

    so based on our findings I assume that you either didn't run yum update after install or your SSH pass is less than 8 non-alphanumeric characters. What kind of customizations are done to your ISO?

    password is 8 char alpha numeric.
    Yes, I probably did not run yum update.
    Its a standard ISO provided by provider.. never had issue in general.

    Not running the yum upgrade will leave your system vulnerable to several exploits depending on the version used. It is absolutely necessary to upgrade for security reasons.

  • @Hoptrics said:

    @plumberg said:

    @Hoptrics said:
    We have 5 honeypot systems configured to evaluate default security configuration in various CentOS, Redhat, Debian, Ubuntu, Mint versions. The only systems that were compromised were due to the following:
    1. SSH root password with less than 8 characters (no special characters)
    2. CentOS 7 base install without running yum updates
    3. Debian 6 base install vulnerabilities (with apt updates)
    4. Ubuntu 16 server base install without apt updates

    so based on our findings I assume that you either didn't run yum update after install or your SSH pass is less than 8 non-alphanumeric characters. What kind of customizations are done to your ISO?

    password is 8 char alpha numeric.
    Yes, I probably did not run yum update.
    Its a standard ISO provided by provider.. never had issue in general.

    Not running the yum upgrade will leave your system vulnerable to several exploits depending on the version used. It is absolutely necessary to upgrade for security reasons.

    It was Centos 8.2. So, I was a little lax here... I usually get to that, but, somehow I left it for the coming weekend to work on... :( and this happened.

  • AlwaysSkintAlwaysSkint Member
    edited September 10

    2 steps that I always do first:
    1. Update = good check for connectivity, amongst other things.
    2. Add authorized_keys and turn off shh password authentication.

    Then it can idle until I setup CSF etc. but usually I just power down until ready to do further installs/configuration. In the time it takes to do those first two steps, I've seen failed login attempts easily exceed a thousand, with some server providers. :o

    Thanked by 1plumberg

    Long live LowEndInfo.com

  • Here's similar case. https://stackoverflow.com/questions/36623596/is-this-file-gcc-sh-in-cron-hourly-malware

    I'm pretty sure there's nothing usefull in /var/log. Propably there's nothing, because they try to hide how they got in.

  • raindog308raindog308 Administrator, Moderator

    @Hoptrics said: CentOS 7 base install without running yum updates 3. Debian 6 base install vulnerabilities (with apt updates) 4. Ubuntu 16 server base install without apt updates

    You're saying that all three of these systems can be compromised remotely by means other than exploiting a week root password?

    For LET support, please visit the support desk.

  • AlwaysSkintAlwaysSkint Member
    edited September 10

    @raindog308 said: ..a week root password?

    Not just once a week. :p
    [Soz]

    Long live LowEndInfo.com

  • Thanks all. Whats a quick easy way to disable all outgoing connections from my VPS?

  • vimalwarevimalware Member
    edited September 11

    Sigh. Everyone please just turn off password auth unless you DO use a 18+char randomly generated password from a tool ( using FULL character set).

    Thanked by 2dosai FootKaput

    250GB USA RAID6 StorageKVM from €13/yr (no-torrenting, TopProvider)
    (affiliate for 🥰 ) https://clients.inceptionhosting.com/aff.php?aff=401&gid=30

  • @vimalware how about fail2ban rather than complicated password???

  • jackbjackb Member, Provider

    @pedagang said:
    @vimalware how about fail2ban rather than complicated password???

    Not effective.

    Thanked by 1pedagang

    Afterburst - Awesome OpenVZ&KVM VPS in US+EU

  • @pedagang said:
    @vimalware how about fail2ban rather than complicated password???

    No. Just no.
    Take 5mins and teach your users about public key crypto.
    private key : your eyes only.
    public key : reqd on all systems you need to talk to.

    Thanked by 2pedagang dfroe

    250GB USA RAID6 StorageKVM from €13/yr (no-torrenting, TopProvider)
    (affiliate for 🥰 ) https://clients.inceptionhosting.com/aff.php?aff=401&gid=30

  • @vimalware said:
    Sigh. Everyone please just turn off password auth unless you DO use a 18+char randomly generated password from a tool ( using FULL character set).

    while I agree that using keys instead of passwords should be the way to go, one should note that this table and given times are kind of meaningless without specification of the crunching setup.
    most likely you will only achieve that, if you're already in possession of the hash so you can bruteforce directly on whatever powerful system.

    the usual attacks against ssh still involve connection limits, latency and such, so it will for sure take a lot longer to brute force even an 8 mixed chars passwords directly. things like fail2ban add to that time frame simply because an attacker then needs to circle through tons of IPs or dial down on the frequency.

    UltraVPS.eu KVM in US/UK/NL/DE: 15% off first 6 month | Netcup VPS/rootDS - 5€ off: 36nc15279180197 (ref)

  • yes, fail2ban still usefull on random attack (little)
    even on default config

    i dont know on masive brute force attack

  • @Falzo said:

    ... this table and given times are kind of meaningless without specification of the crunching setup.
    most likely you will only achieve that, if you're already in possession of the hash so you can bruteforce directly on whatever powerful system.

    Yes you're right. I have posted the wrong image for this context.
    This is for hashed password dumps.

    Thanked by 1Falzo

    250GB USA RAID6 StorageKVM from €13/yr (no-torrenting, TopProvider)
    (affiliate for 🥰 ) https://clients.inceptionhosting.com/aff.php?aff=401&gid=30

  • plumbergplumberg Member
    edited September 11

    Thank you all. Atleast for now I know what was the malware. But I haven't figured out anything from logs.

    So many people suggest Key based authentication...

    How does that work when I have multiple devices to login from? Like my phone/ home laptop/ work laptop/ an idling vps... what's best way to setup keys? Also if one pvt key on a device is compromised, wouldn't the attacker be able to remove other keys from server?

    Thank you

  • key900key900 Member, Provider

    @plumberg said:
    Thank you all. Atleast for now I know what was the malware. But I haven't figured out anything from logs.

    So many people suggest Key based authentication...

    How does that work when I have multiple devices to login from? Like my phone/ home laptop/ work laptop/ an idling vps... what's best way to setup keys? Also if one pvt key on a device is compromised, wouldn't the attacker be able to remove other keys from server?

    Thank you

    Just use termius it will rsync all ssh keys with all devices.

    Regards

    Thanked by 2pedagang TimboJones
    *LetBox.Com Make it Simple, Dallas & Los Angeles - Ryzen Pure NVMe + Block Storage Boxes!. Discord Community https://discord.gg/g6dqjmm
  • @key900 said:

    @plumberg said:
    Thank you all. Atleast for now I know what was the malware. But I haven't figured out anything from logs.

    So many people suggest Key based authentication...

    How does that work when I have multiple devices to login from? Like my phone/ home laptop/ work laptop/ an idling vps... what's best way to setup keys? Also if one pvt key on a device is compromised, wouldn't the attacker be able to remove other keys from server?

    Thank you

    Just use termius it will rsync all ssh keys with all devices.

    Regards

    I'll check. Thnx

  • Don't mean to hijack a topic but out of curiosity is host.deny host.allow configured to allow connection from just one IP enough to keep them out or are there any work arounds for it?

    I swear to drunk Im not god

  • @Falzo said:
    while I agree that using keys instead of passwords should be the way to go, one should note that this table and given times are kind of meaningless without specification of the crunching setup.
    most likely you will only achieve that, if you're already in possession of the hash so you can bruteforce directly on whatever powerful system.

    the usual attacks against ssh still involve connection limits, latency and such, so it will for sure take a lot longer to brute force even an 8 mixed chars passwords directly. things like fail2ban add to that time frame simply because an attacker then needs to circle through tons of IPs or dial down on the frequency.

    You are totally right. Yes, passwords are bad, but not that bad.
    Lets say its a 8 character password with lower- and uppercase plus numbers. That's 62 possible characters, that gives 62^8 possible combinations.
    Just to keep the math easy we say one attempt per second.
    62^8 combinations = 218 trillion attempts. With 1 attempt per second it would take about 7 million years to guarantee that the password is found. We can go extreme and say that it manages 1000 attempts per second. Still, it takes 7000 years.
    And simple things like fail2ban and similar multiplies that time by a lot. Such a simple thing as not permitting root login forces the attacker to have to guess the username as well as the password. Anyone with basic understanding of math can figure out what that does to the total time taken (unless the user is an idiot and uses 'admin' or similar as username).

    Don't get me wrong, I'm not advocating using just passwords, absolutely not, but if people could just use good, random passwords bruteforce attacks would be more or less useless. There are of course a lot of other reasons to not use just passwords, but bruteforce against services such as ssh is so easily prevented it shouldn't even be a problem.
    And again, since I know I will probable take some heat for this: I'm not advocating using passwords! I just think people should understand what an extremely big difference it makes to just add a few numbers or symbols in their passwords.

    Thanked by 2Falzo chocolateshirt
  • @rcy026 said: Such a simple thing as not permitting root login forces the attacker to have to guess the username as well as the password

    totally agree on that one. if you install debian/ubuntu via netinst or as minimal, root access will be set to no-password as this is the openssh-server default - while you'll still get an unprivileged user to be able to login with a password and become root via su - afterwards.

    it's only a convenience/automation thing nowadays that has providers still setup templates with root access via password enabled, so they can simply set something and send to the user 🤷‍♂️

    UltraVPS.eu KVM in US/UK/NL/DE: 15% off first 6 month | Netcup VPS/rootDS - 5€ off: 36nc15279180197 (ref)

  • noted
    root via su
    like one of my provider

Sign In or Register to comment.