Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Advertise on LowEndTalk.com
Server hacked? Extrem outgoing Traffic
New on LowEndTalk? Please Register and read our Community Rules.

Server hacked? Extrem outgoing Traffic

Hello community,
I have a "small" problem and I have Proxmox on a Hetzner server, with which I am currently learning everything about Debian and virtualization as a hobby.
Apparently my server has now been hacked because I have such high outgoing traffic all the time that I can hardly connect to the server.
The root password was also different, which I was able to change again thanks to the rescue system.
Unfortunately, I'm not far enough to know how to recognize where it all comes from and how to block it. That's why I thought to ask you professionals. Unfortunately, Hetzner does not offer any help.
I hope I have described my problem in enough detail. Probably the whole thing happened through outdated software.

Kind regards and thank you in advance for your tips and help.

Comments

  • JioJio Member

    Wipe. Now.

  • dustincdustinc Member, Provider

    It does sound like your server was compromised, it can be from a number of things (most commonly we see the reasoning to be a weak root password).

    As @Jio recommended, reinstall and start fresh assuming you have no important data to extract prior to doing so.

    RackNerd LLC - Introducing Infrastructure Stability
    Dedicated Servers, Private Cloud, DRaaS, Colocation, VPS, DDoS Mitigation, Shared & Reseller Hosting

  • NeoonNeoon Member
    edited November 10

    Well, spiceproxy which ships by default with proxmox enabled hat a RCE.
    https://security-tracker.debian.org/tracker/CVE-2020-14355
    https://www.cybersecurity-help.cz/vdb/SB2020100911

    Didn't you patch?
    I disable it by default + firewall it.

    "Make Proxmox Premium again!"
    https://gist.github.com/Ne00n/17e579b1082115da15afbc5c65c9cbf7

    Thanked by 1dahartigan
  • Use SSH key authentication and disable password login.

  • @msallak1 said:
    Use SSH key authentication and disable password login.

    If it was hacked, they'd be silly not to install their own key at the very least.

    It's fucking wipe and wipe only. It was compromised and needs to be put down.

  • dahartigandahartigan Member without signature

    Check in your etc shadow file for any extra users that may be in there. You can tell because you'll see root with a very long string, and any other users that YOU added will also have the same thing.

    If you see any users with that long string (which is a password hash), and they aren't root or a user you made, then you've got a problem.

    Thanked by 1SnowStylez
  • If it was hacked ...
    It's fucking wipe and wipe only. It was compromised and needs to be put down.

    Unless it was an unprivileged user that was hacked.

    Had a user with a weak shared password on one of my servers get someone in not long ago (going by the logs it appeared that attacker was using a list of account names and passwords picked up from hacks elsewhere, though it could have also been a more general brute force attack). The attackers could do very little to the machine overall (caveat: if you are behind with updates this might not always be true: privilege escalation bugs could allow root access for otherwise locked down accounts) but were able to spin the CPU and spew out garbage as that lowly user (it looked to be a mining drone, and the outgoing traffic was targetting a mix of ports 22 & 443 so it was presumably trying to spread itself via other weak accounts accessible via SSH and web based control panels).

    But I would agree if a directly privileged user, or one with sudo-with-no-password access, was compromised. There is no telling what extra backdoors the exploit could have added to allow reentry after you clean up the first attack.

  • First of all, thank you for all your answers and tips. I would like to temporarily solve the problem in order to get backups, since individual things like Nextcloud and Plex were running productively but isolated from the host system (since virtualized + encrypted). My idea would be to block the ports that cause the traffic or the IP. Unfortunately, I just don't know how. Then I would set up the server again immediately and set it up again according to the usual security measures. I am very satisfied with the solution at Hetzner because I can use the server well for my studies. But my laziness has shown me what can happen, you learn from mistakes. Again I won't let it get that far. That he was hacked was only a guess in the history because of the root password, I couldn't find anything else. Users have also stayed the same.

  • marvelmarvel Member, Provider

    Yep full wipe is your only option. Rather sooner than later.

  • @marvel said:
    Yep full wipe is your only option. Rather sooner than later.

    But how can I temporarily solve the problem in order to take backups?

  • marvelmarvel Member, Provider

    @Tfouabi said:

    @marvel said:
    Yep full wipe is your only option. Rather sooner than later.

    But how can I temporarily solve the problem in order to take backups?

    Can you temporary disconnect it from internet and access it from the KVM console? (Don't know if Hetzner supports it).

    Was a VM hacked or the hypervisor, do your VMs have public IPs? I would shut down all VMs first, see if that fixes it.

  • Like others have said, you have to wipe it immediately and reinstall. I also disabled SSH access on IPv4, if you have IPv6 generate an IPv6 address and use it only for SSH access.

    ListenAddress 6c5e:90cf:c36a:a638:9123:237e:91ab:b17b

  • Even if you were to get the data off, can you trust it?

  • @Tfouabi said:

    @marvel said:
    Yep full wipe is your only option. Rather sooner than later.

    But how can I temporarily solve the problem in order to take backups?

    You need to firewall the bad traffic. But a) you need to know what to block, and b) you need to know how to block it in your firewall

Sign In or Register to comment.