Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Possible server break-in. What are my options?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Possible server break-in. What are my options?

Hi there,

I fear that today, after nearly 10 years of taking care of my own servers, I became victim of a server break-in. I don't know how it happened yet, but it seems as if somebody was able login to one of my servers with a Webmin/Virtualmin installation and had a good time deleting random data in the /home directory. It may also be possible that there was some other fault, but I cannot imagine which one. The server was running without issues for more than a year and received all system and software updates on a daily basis. Access via SSH was only possible via SSH keys (no passwords). SSH is not running on a standard port and so is the Webmin/Virtualmin panel. I was the only person to access the server and having the necessary credentials.

What would be the right course of action now?

I will check my computers for possible malware/trojans, of course. But what "forensic" possibilities do I have to find out what might have happened? I never came across a situation like this and am just an ambitious amateur. I will re-install the server from scratch after some investigation, but I would really like to find out, how that was possible.

I have backups of all data, so no problem there and the server is powered off for the moment.

Thank you in advance for any helpful hints!

Thanked by 1sambling

Comments

  • Do you have an IDS by any chance?

    Thanked by 13606202
  • FHRFHR Member, Host Rep
    edited October 2018

    Logs will tell you a lot. Check for unknown processes running and recent file changes/new files (find with time parameters)

    If you wanted to do full forensics, you'd want to take a memory dump and hard drive image first - but that is probably unnecessary. And seeing as you already powered down the server, taking a memory dump after a fresh boot would not help anyway.

    You want to find out answers to these:

    • Did the attacker go in through some web app you host, through the panel itself, or through a running service?
    • Are any new services running / communicating via network?
    • Which files changed?
    • Are there known vulnerabilities for software versions you use?
    • If you have monitoring (with graphs), did any value suddenly change? (CPU utilization, disk utilization, network utilization etc)
    • Which privileges did the attacker manage to get? Root, or just some service user?
    • Is there anything in crontab / .bash_history of all users?
    • Is there anything suspicious in logs before AND after the break-in?
    Thanked by 33606202 default MasonR
  • You told us what measures you took to prevent hacking but didn't tell us which you didn't. It's not what you know is dangerous but what you don't know.

    I assume webmin was used to host something to the outside world. Maybe your site had an exploit.

    Thanked by 13606202
  • 36062023606202 Member
    edited October 2018

    FlamesRunner said: Do you have an IDS by any chance?

    No, not really. There is CSF/LFD running on that server and the built-in brute force protection that comes with Webmin/Virtualmin and it is configured to be quite strict. But I suspect that both does not count as Intrusion Detection System.

    FHR said: Logs will tell you a lot. Check for unknown processes running and recent file changes/new files (find with time parameters)

    If you wanted to do full forensics, you'd want to take a memory dump and hard drive image first - but that is probably unnecessary. And seeing as you already powered down the server, taking a memory dump after a fresh boot would not help anyway.

    You want to find out answers to these:

    Thank you very much for the detailed list of things to check. I will spend my weekend now going through your valuable advices!

    Yura said: You told us what measures you took to prevent hacking but didn't tell us which you didn't. It's not what you know is dangerous but what you don't know.

    I assume webmin was used to host something to the outside world. Maybe your site had an exploit.

    I agree. I am still a bit shocked and surprised about that incident and will go through the four websites that are hosted on the server. There is no CMS installed, everything facing the outside world is plain HTML. But this is only what I think, I will have to verify that during the next hours. As you said, "It's not what you know is dangerous but what you don't know."...

  • By any chance the deleted stuff is in fact all the files under a virtual account?
    If yes, do you use virtualmin built-in backup?
    If yes, did this happen after a failed backup ?

    I've faced this problem and it always happens after a failed backup, but honestly I never pursuit it much.

    Thanked by 13606202
  • 404error said: By any chance the deleted stuff is in fact all the files under a virtual account?

    Indeed, it seems as if all missing data is just under one single virtual account. But not all data is missing. Only 2 complete directories, the rest seems intact.

    404error said: If yes, do you use virtualmin built-in backup?

    Yes, I do.

    404error said: If yes, did this happen after a failed backup ?

    No. At least not as far as I can see. I get an eMail every time that the daily backups runs and it never reported a problem. That does not mean that there wasn't any, but the report was always positive.

    404error said: I've faced this problem and it always happens after a failed backup, but honestly I never pursuit it much.

    That seems like quite a bug then. I was thinking about switching to cPanel anyway, but I did not see the need to pay so much every month for a personal server without selling services. Webmin/Virtualmin did a good job for me during the last years, but if there is a quirk, then it's extraordinary annoying...

    Again, thank you all! This has all been very helpful!

  • 3606202 said: No. At least not as far as I can see. I get an eMail every time that the daily backups runs and it never reported a problem. That does not mean that there wasn't any, but the report was always positive.

    Did you read the whole email? some virtual hosts may have no errors and some may have an error. The same email will have them all.

    3606202 said: That seems like quite a bug then. I was thinking about switching to cPanel anyway, but I did not see the need to pay so much every month for a personal server without selling services. Webmin/Virtualmin did a good job for me during the last years, but if there is a quirk, then it's extraordinary annoying...

    Well, I guess it depends on how important are those projects for you, in my case, the nightly backups are enough and I can handle a couple hours downtime.
    So I never looked into it (I should have, I know) and at the same time I don't think it's worth paying for cPanel. As for free panels, webmin & virtualmin still feel as the easiest, feature rich , free panel around.

    As a sidenote, I've been using Webmin for quite a long time, and this issue only happened a few times.

    Thanked by 13606202
  • I thought it was standard procedure to always wipe the o/s if you suspect a break in and copy files over that does not execute ... what changed?

    Thanked by 1birchbeer
  • FHRFHR Member, Host Rep

    @smile said:
    I thought it was standard procedure to always wipe the o/s if you suspect a break in and copy files over that does not execute ... what changed?

    Ideally you want to investigate how the attacker got in in the first place. Reinstall would be just a temporary solution if you make the same config mistake/install the same vulnerable software/use same compromised password or whatever - the attacker would just get in again.

    Thanked by 13606202
  • 36062023606202 Member
    edited October 2018

    Like FHR said...

    However, I did as much research as I could during the weekend.

    • The relevant logs show nothing suspicious.
    • There are no new services running that interact with the network and no new services in general.
    • I did a comparison of the files via their checksum towards a 6 month old backup. The data has practically not changed since then (despite some small content that I added myself). The checksums are identical. Either something was already changed by an attacker 6 months ago, or nothing like that happened.
    • Munin graphs show absolutely no change in CPU/RAM/Network/Disk usage.
    • I could not find any altered or suspicious new cron job.

    It may really have been the Virtualmin backup bug that @404error mentioned. The backup logs of the last 14 days do not report any malfunction, but who knows. Strange thing.

    I will give up. Whatever happened - I won't find out. I have now setup the server from scratch, all credentials changed and I copied the /home directory over from the backup. I will keep an eye on it, but I guess the haunting is over.

    However - Does anybody have a good (and possibly free) IDS to recommend that works well on Debian? Maybe I should install one to make the monitoring easier...

    Again, thank you for all your help, this was very much appreciated!

  • birchbeerbirchbeer Member
    edited October 2018

    @3606202 said:
    However - Does anybody have a good (and possibly free) IDS to recommend that works well on Debian? Maybe I should install one to make the monitoring easier...

    Security Onion is a good place to start - https://securityonion.net/ It's a suite of multiple solutions.

    I normally recommend Suricata - https://suricata-ids.org/ if you want to just use a point solution.

    Bear in mind that an IDS against a persistent adversary is not a silver bullet.

  • @3606202 said:
    However - Does anybody have a good (and possibly free) IDS to recommend that works well on Debian? Maybe I should install one to make the monitoring easier...

    I just finished a tutorial on how to "misuse" HAProxy to detect TCP port scans. I happen to use Debian, too. It probably won't take you more than 10 minutes to set it up.

    http://botnet-tracker.blogspot.com/2018/10/tcp-port-scan-detection-with-haproxy.html

    Thanked by 1pxhaxor
  • @birchbeer

    I think that might be a little too much for the op. Even tho you have posted a free solution (the second one). Is probably that backup bug that was mentioned around.

    @3606202

    Did you had backups? Was it sensitive data? Is probably easier to wipe the server and start again, if possible don't use any panels, unless is something like Centminmod which is all CLI.

    A good rule: If using any web control panel go with cPanel or Plesk.

    @birchbeer said:

    @3606202 said:
    However - Does anybody have a good (and possibly free) IDS to recommend that works well on Debian? Maybe I should install one to make the monitoring easier...

    Security Onion is a good place to start - https://securityonion.net/ It's a suite of multiple solutions.

    I normally recommend Suricata - https://suricata-ids.org/ if you want to just use a point solution.

    Bear in mind that an IDS against a persistent adversary is not a silver bullet.

  • @Hxxx said:
    I think that might be a little too much for the op. Even tho you have posted a free solution (the second one). Is probably that backup bug that was mentioned around.

    Maybe... The OP's setup and technical proficiency wasn't clear to me. I was just offering 2 free suggestions which are commonly used since OP asked for free IDS solutions and indicated that the server has been rebuilt from scratch with new credentials. An IDS may aid in forensics if something like this occurs in future or possibly even detect an attack.

    I agree it could be a defect in virtualmin but it's unusual for files to be unlinked during a backup.

Sign In or Register to comment.