Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Vulnerability in SolusVM Debian 10 template - "debianuser" backdoor/default user
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Vulnerability in SolusVM Debian 10 template - "debianuser" backdoor/default user

Daniel15Daniel15 Veteran
edited May 2021 in General

From another thread:

@redgreenblue said:
Anyone with VM with Hosthatch running with Debian image provided by Hosthatch should check their VM now. Good probability is that your VM may have been compromised via user debianuser, which is now running cnrig, which appears to be related to CryptoNight. Hosthatch VMs in multiple locations has been observed to have been compromised this way.

From the provider via email:

We have detected a security vulnerability in our Debian 10 template and our records indicate that you have installed a VM with this template. If you have since then reinstalled your VM to any template other than Debian 10, or used an ISO to reinstall your VM, you can ignore this email.

...

How could this happen?
We use SolusVM as our backend virtualization platform, it is a leading provider operated by Plesk. We are using their official templates. Unfortunately this particular template had an issue which resulted in this security vulnerability. They are aware of the situation.

How was it fixed?
We have patched the template with help from SolusVM and they also helped us to confirm that no other templates are affected.

I also found this Chinese blog post from October 2020, where someone's GreenCloudVPS VPS was compromised through what I assume is the same debianuser account, also running some other crypto thing (xmrig): https://aoyouer.com/posts/server-hacked-record.html, so this has been in the wild for at least four months (probably longer), and likely affects many other providers too

Please check your servers for a debianuser user. If so, you're probably best off wiping the whole thing and restoring from backups.

You should be fine if password authentication is disabled, as in that case you can only access SSH if you have the private key. I'd still recommend deleting the debianuser user if it's present on your system.

If you still use password authentication for SSH, I'd strongly recommend:

  1. Generate an SSH key. You may already have one if you use a service like GitHub that uses SSH keys for authentication. If you don't have one already, an Ed25519 key is good. https://medium.com/risan/upgrade-your-ssh-key-to-ed25519-c6e8d60d3c54
  2. Ensure the key is in ~/.ssh/authorized_keys
  3. Disable PasswordAuthentication in /etc/ssh/sshd_config and restart SSH (service ssh restart)
  4. Double-check that you can still get in (open a new session and test it out) before you exit your active SSH session
«1345

Comments

  • DPDP Administrator, The Domain Guy

    @dustinc - Where do you get your templates from? I checked my Racknerd node, running Buster, and I can acknowledge the existence of the debianuser account.

    Thanked by 1lokuzard
  • I guess I'd never use the templates again. I saved a few minutes by using the templates, now I have to spend hours to install all my stuff all over again.

  • NeoonNeoon Community Contributor, Veteran
    edited February 2021

    Can confirm it for Virmach, has no ssh key, so just password I guess.

    Thanked by 1pbx
  • jarjar Patron Provider, Top Host, Veteran
    edited February 2021

    I'm pretty sure I recall dealing with a similar issue at a larger cloud provider that doesn't use those templates, but the memory has faded.

    Thanked by 1miu
  • Oh Jesus. I came across this issue on one of my idlers a few months back. Nothing was compromised except they ran a miner.

    Always check etc passwd for unknown users and for extra points check etc shadow for entries with a hash, there should only ever be your user and/or root.

    Thanked by 2Daniel15 VPSforVPN
  • WebProjectWebProject Host Rep, Veteran

    @Kiwi83 said:
    I guess I'd never use the templates again. I saved a few minutes by using the templates, now I have to spend hours to install all my stuff all over again.

    To install from ISO doesn’t take hours, max 15-20 mins and it will be exactly as you required, instead of default settings.

    What do you expect it’s SolusVM? As is out of date now with features wise, as normally any templates will ask to set your own root password.

  • Ah good reminder that I should not be using templates, but unfortunately I always forget. I’d like to think I’m probably safe as I deleted the debianuser and disabled password authentication long ago, but without knowing the exact vulnerability I’m not feeling very safe.

    Thanked by 1pbx
  • dustincdustinc Member, Patron Provider, Top Host

    @thedp said:
    @dustinc - Where do you get your templates from? I checked my Racknerd node, running Buster, and I can acknowledge the existence of the debianuser account.

    Hi @thedp -- we obtain our templates through tdn.solusvm.com -- we too are investigating this. Thank You!

  • Has there been sufficient checking on other templates?

    I bet all those raged out butthurt idlers that were suspended and didn't know why are having an "oh" moment.

    Thanked by 2t0m coreflux
  • So I guess asking what people are doing tonight is rhetorical. I presume everyone is reinstalling Debian and using the ISO.

    Thanked by 2yoursunny VPSforVPN
  • DPDP Administrator, The Domain Guy

    @dustinc said:

    @thedp said:
    @dustinc - Where do you get your templates from? I checked my Racknerd node, running Buster, and I can acknowledge the existence of the debianuser account.

    Hi @thedp -- we obtain our templates through tdn.solusvm.com -- we too are investigating this. Thank You!

    https://templates.solusvm.com/ ? :joy:

    Thanked by 1coreflux
  • @its420somewhere said: Oh Jesus. I came across this issue on one of my idlers a few months back. Nothing was compromised except they ran a miner.

    IMO it's better to keep the machine shutdown when not in use: this kind of problem becomes impossible, and the RAM of the host can be put to use for people who might actually use it.

  • @TimboJones said:
    Has there been sufficient checking on other templates?

    I bet all those raged out butthurt idlers that were suspended and didn't know why are having an "oh" moment.

    I'm guessing @virmach is going to have a large drop in server load and same with @hosthatch and racknerd

  • FranciscoFrancisco Top Host, Host Rep, Veteran

    @TimboJones said: Has there been sufficient checking on other templates?

    I expect Ubuntu ones to have a ubuntu user or similar since they block root login by default, like Debian 10.

    It takes some tinkering on preseed files to deal with that.

    Francisco

  • dustincdustinc Member, Patron Provider, Top Host

    @thedp said:

    @dustinc said:

    @thedp said:
    @dustinc - Where do you get your templates from? I checked my Racknerd node, running Buster, and I can acknowledge the existence of the debianuser account.

    Hi @thedp -- we obtain our templates through tdn.solusvm.com -- we too are investigating this. Thank You!

    https://templates.solusvm.com/ ? :joy:

    @thedp
    It looks like http://tdn.solusvm.com/ only displays the recent/popular OS templates, and https://templates.solusvm.com/kvm/ has an exhaustive list of all available KVM templates, even ones that previously displayed on the TDN and are no longer displaying.

    As an FYI we are working on implementing a patch for this at the moment.

  • Daniel15Daniel15 Veteran
    edited February 2021

    I wonder if any LXC or OpenVZ templates have a similar issue... Those environments are tricker since you can't just install from ISO, so you have to trust that the templates are legit (or manually audit etc passwd and remove any unexpected entries).

    @WebProject said:

    @Kiwi83 said:
    I guess I'd never use the templates again. I saved a few minutes by using the templates, now I have to spend hours to install all my stuff all over again.

    To install from ISO doesn’t take hours, max 15-20 mins and it will be exactly as you required, instead of default settings.

    I think they mean it'll take them hours to get everything back into the same state - Wipe the system, restore backups (potentially from some other storage VPS provider with slow disks and/or network), reinstall all the packages they use, etc).

    @TimboJones said:
    Has there been sufficient checking on other templates?

    I bet all those raged out butthurt idlers that were suspended and didn't know why are having an "oh" moment.

    I was thinking the same thing! I wonder how many idlers are actually mining crypto or whatever.

    @pbx said:

    @its420somewhere said: Oh Jesus. I came across this issue on one of my idlers a few months back. Nothing was compromised except they ran a miner.

    IMO it's better to keep the machine shutdown when not in use: this kind of problem becomes impossible, and the RAM of the host can be put to use for people who might actually use it.

    Maybe I'm in the minority but I hate when providers boot systems as soon as they're provisioned. Sometimes I get VPSes from European providers and they boot them while I'm asleep. I don't want the system to be booted until I've got time to manually install the OS from an ISO.

    Thanked by 3WebProject pbx t0m
  • DPDP Administrator, The Domain Guy
    edited February 2021

    I think it's worth tagging all the providers here so they are aware of this and can act on it immediately.

    Provided that mods/admins agree and will do it.

    @raindog308 @hzr

  • @thedp - it will be interesting to see how many providers actually do anything about it. I am presuming there will be more than a few that do nothing

    Thanked by 1DP
  • DPDP Administrator, The Domain Guy

    @Unbelievable said:
    @thedp - it will be interesting to see how many providers actually do anything about it. I am presuming there will be more than a few that do nothing

    Security is a priority and should go above most things.

    But yeah, it's up to them if they want to act on it or not but at least we've done our part to highlight it to them because that would be the right thing to do :)

  • DPDP Administrator, The Domain Guy

    @dustinc said: As an FYI we are working on implementing a patch for this at the moment.

    You should also send out comms about this in parallel.

  • a patch? shouldnt people just wipe out the install and use the iso to be certain rather than trusting a firm with its first extremely large security incident (yes, not of their making) can successfully patch something?

  • FranciscoFrancisco Top Host, Host Rep, Veteran

    Instead of trying to fix the EXT4 issues (debian uses a newer version of EXT4 that CentOS 7 can't handle) they just...use EXT3 instead.

    God bless SolusVM.

    Francisco

  • dustincdustinc Member, Patron Provider, Top Host

    @Unbelievable said:
    a patch? shouldnt people just wipe out the install and use the iso to be certain rather than trusting a firm with its first extremely large security incident (yes, not of their making) can successfully patch something?

    Existing customers should reinstall, that is the right thing to do. As for the patch, it's for the template, for new reinstallations going forward.

  • dustincdustinc Member, Patron Provider, Top Host

    @thedp said:

    @dustinc said: As an FYI we are working on implementing a patch for this at the moment.

    You should also send out comms about this in parallel.

    Absolutely, we're transparent and will be sending communication regarding this.

    Thanked by 1DP
  • Why not just kill the template and tell new people to use the iso? From a risk abatement perspective- in my mind thats the most prudent thing to do. Or do you have a robust workflow for creating patches and testing them that has been tested for efficacy? Just curious as some companies have more robust engineering that others

  • jackbjackb Member, Host Rep
    edited February 2021

    @Unbelievable said:
    Why not just kill the template and tell new people to use the iso? From a risk abatement perspective- in my mind thats the most prudent thing to do. Or do you have a robust workflow for creating patches and testing them that has been tested for efficacy? Just curious as some companies have more robust engineering that others

    Most customers find installing from ISO annoying, so template installs are important to offer.

    I've never had much faith in solus's tdn (the modified/uploaded date alone should cause some concern), so we built our own build server and we spend 30 mins making new templates when needed. Build server takes care of keeping it fresh.

    We're definitely not alone in that category -- I can think of several providers here who don't trust the tdn and roll their own templates.

  • @Unbelievable said:

    @TimboJones said:
    Has there been sufficient checking on other templates?

    I bet all those raged out butthurt idlers that were suspended and didn't know why are having an "oh" moment.

    I'm guessing @virmach is going to have a large drop in server load and same with @hosthatch and racknerd

    Maybe not so much Virmach if they've been catching the worst already.

  • dustincdustinc Member, Patron Provider, Top Host

    @Unbelievable said:
    Why not just kill the template and tell new people to use the iso? From a risk abatement perspective- in my mind thats the most prudent thing to do. Or do you have a robust workflow for creating patches and testing them that has been tested for efficacy? Just curious as some companies have more robust engineering that others

    @Unbelievable
    I understand where you're coming from. Most of our customers who are more tech-savvy (ones from this community I noticed) tend to install from ISO's, but we also have customers who prefer the convenience of a template, for the sake of the one click reinstalls. Also, developers often use OS templates for quick reinstall convenience, as it allows them to quickly change/swap over to a different OS without having to go through the manual OS reinstallation process each time. In either case, we'll be notifying our customers so they can choose whether they wish to install from an ISO manually, or if they'll be reinstalling from the updated Debian 10 template.

  • From what I can see, the debianuser account has a weak password and that's the thing that needs patching. Kill the account, kill the vuln. You will know if you were hit because your CPU will be at full throttle. If you have not got xmrig running, you weren't hit. Remove the debianuser.

    If you have xmrig running and were compromised, reinstalling is the only safe option.

    Thanked by 1pbx
  • DPDP Administrator, The Domain Guy

    Also, maybe this thread should be pinned for a while?

    Thanked by 2saibal Daniel15
Sign In or Register to comment.