Vulnerability in SolusVM Debian 10 template - "debianuser" backdoor/default user

Razza

@Daniel15 said:
Am I seeing things right? They kept a copy of the original version and just added a hyphen to the end of the name (etc/passwd-)? wut.

password- is just a automated backup the system makes, even Shadows got a - version.

Emil

@udonworld said:
Where is the announcement by SolusVM?

The Debian 10 images linux-debian-10-x86_64-gen2-{v1,v2}.gz on templates.solusvm.com were apparently updated very recently - without even changing the filename. This alone doesn't smell good.

Anyway, I downloaded the latest image locally to find out how they "patched" it... Geez, they just mounted the image and edited /etc/passwd and /etc/shadow instead of recreating the image from scratch!?

$ ls -l /media/etc/{passwd,shadow}*
-rw-r--r-- 1 root root   1346 Feb  2 20:03 /mnt/etc/passwd
-rw-r--r-- 1 root root   1343 Oct 21  2019 /mnt/etc/passwd-
-rw-r----- 1 root shadow  808 Feb  2 20:03 /mnt/etc/shadow
-rw-r----- 1 root shadow  913 Oct 21  2019 /mnt/etc/shadow-
$ grep debianuser /media/etc/passwd
$ grep debianuser /media/etc/passwd-
debianuser:x:1000:1000:DebianUser,,,:/home/debianuser:/bin/bash
$ ls -l /media/home
total 4
drwxr-xr-x 2 k k 4096 Oct 21  2019 debianuser

I believe I was the first, or at least among the first, to notify SolusVM about this and I have been pressing them to understand the severity and announce it to all their customers. The only response though was:

The templates were updated at https://templates.solusvm.com/kvm Also, we prepared the article regarding this vulnerability and steps to deal with it here: https://support.solus.io/hc/en-us/articles/360019368659

Except from that article (which is very hard to find without having the URL), I am not aware of any announcement.

Daniel15

@Emil said:

@udonworld said:
Where is the announcement by SolusVM?

I believe I was the first, or at least among the first, to notify SolusVM about this and I have been pressing them to understand the severity and announce it to all their customers. The only response though was:

> The templates were updated at https://templates.solusvm.com/kvm > Also, we prepared the article regarding this vulnerability and steps to deal with it here: > https://support.solus.io/hc/en-us/articles/360019368659 >

Except from that article (which is very hard to find without having the URL), I am not aware of any announcement.

Their article basically just copied this post. The last sentence in the "for existing VPSes" section ("Double-check that you can still get in (open a new session and test it out) before you exit your active SSH session") is identical to my last sentence in this post, and they linked to the same Ed25519 article I linked to. That obviously means they've seen this post.

They also completely missed the part about checking whether the vulnerability was actually exploited - their steps are just to delete the user and configure an SSH key. No checking of logs or anything.

DP

@Daniel15 said:
Their article basically just copied this post. The last sentence in the "for existing VPSes" section ("Double-check that you can still get in (open a new session and test it out) before you exit your active SSH session") is identical to my last sentence in this post, and they linked to the same Ed25519 article I linked to. That obviously means they've seen this post.

They also completely missed the part about checking whether the vulnerability was actually exploited - their steps are just to delete the user and configure an SSH key. No checking of logs or anything.

@Francisco summed it up nicely.

@Francisco said: That is a straight up "We think we're going to get sued so we'll silent patch this and hope no one calls us on our bullshit".

MikePT

@Daniel15 said:

@Emil said:

@udonworld said:
Where is the announcement by SolusVM?

I believe I was the first, or at least among the first, to notify SolusVM about this and I have been pressing them to understand the severity and announce it to all their customers. The only response though was:

> > The templates were updated at https://templates.solusvm.com/kvm > > Also, we prepared the article regarding this vulnerability and steps to deal with it here: > > https://support.solus.io/hc/en-us/articles/360019368659 > >

Except from that article (which is very hard to find without having the URL), I am not aware of any announcement.

Their article basically just copied this post. The last sentence in the "for existing VPSes" section ("Double-check that you can still get in (open a new session and test it out) before you exit your active SSH session") is identical to my last sentence in this post, and they linked to the same Ed25519 article I linked to. That obviously means they've seen this post.

They also completely missed the part about checking whether the vulnerability was actually exploited - their steps are just to delete the user and configure an SSH key. No checking of logs or anything.

I can't believe they didn't email all their clients. What a clusterfuck. This is a MAJOR security issue they won't care to tell to their customers!

Neoon

MikePT

@udonworld said:
Where is the announcement by SolusVM?

The Debian 10 images linux-debian-10-x86_64-gen2-{v1,v2}.gz on templates.solusvm.com were apparently updated very recently - without even changing the filename. This alone doesn't smell good.

Anyway, I downloaded the latest image locally to find out how they "patched" it... Geez, they just mounted the image and edited /etc/passwd and /etc/shadow instead of recreating the image from scratch!?

$ ls -l /media/etc/{passwd,shadow}*
-rw-r--r-- 1 root root   1346 Feb  2 20:03 /mnt/etc/passwd
-rw-r--r-- 1 root root   1343 Oct 21  2019 /mnt/etc/passwd-
-rw-r----- 1 root shadow  808 Feb  2 20:03 /mnt/etc/shadow
-rw-r----- 1 root shadow  913 Oct 21  2019 /mnt/etc/shadow-
$ grep debianuser /media/etc/passwd
$ grep debianuser /media/etc/passwd-
debianuser:x:1000:1000:DebianUser,,,:/home/debianuser:/bin/bash
$ ls -l /media/home
total 4
drwxr-xr-x 2 k k 4096 Oct 21  2019 debianuser

I guess they updated it again, can't find those two files, passwd- / shadow-, what a clusterfuck.

Edit: But they forgot to remove /home/debianuser

Edit: File gshadow has it still:

debianuser:!::

What an incompetence.

rm_

But they forgot to remove /home/debianuser

They are reading this right now for hints where else to remove

DP

If that's all they're communicating, they might as well just reference this thread

ben47955

Hello Solus tech !

MikePT

@rm_ said:

But they forgot to remove /home/debianuser

They are reading this right now for hints where else to remove

Yeah I guess we need to do their job :P

Neoon

Inform anyone that may use SolusVM, surely, enough people will find each other to pull SolusVM to court.
Or even better, boycott SolusVM as costumer.

TimboJones

I don't have any of these miners, but I'm curious if there's a way to determine who is getting paid, and whether one could just repoint it to their own wallet.

entrailz

@TimboJones said:
I don't have any of these miners, but I'm curious if there's a way to determine who is getting paid, and whether one could just repoint it to their own wallet.

Usually any miner like xmrig/cnrig either have a config file with the wallet address and pool, in the cli params or they'll compile it with the config. If someone has the miner etc, it might be worth contacting the pool operator and letting them know the address so they can freeze future payouts for the user, making their operation useless.

VirMach

@Unbelievable said: I'm guessing @virmach is going to have a large drop in server load and same with @hosthatch and racknerd

I don't believe this issue is widespread. While the vulnerability definitely exists, I do not believe many customers were compromised. We have systems in place that basically block, to some low level, anyone going around trying to log into people's servers.

This is something that would definitely have a huge impact if it were widespread, with the hackers running a crypto miner.

@Daniel15 said: I wonder if any LXC or OpenVZ templates have a similar issue...

We did have an issue with people's services on OpenVZ, to some degree, having xmrig, but most of these users when we dove in and handled the case, if not nearly all, had an insecure root password or the customer was deliberately mass mining.

@Daniel15 said: so this has been in the wild for at least four months (probably longer)

This has most likely been around since at least October 2019.

@MikePT said: Has SolusVM emailed the providers about this?

@udonworld said: Where is the announcement by SolusVM?

@Neoon said: Inform anyone that may use SolusVM, surely, enough people will find each other to pull SolusVM to court.
Or even better, boycott SolusVM as costumer.

I don't believe they notified customers, nor would I see them doing that. The last time we had a close call we had to have someone locate it, and then we had to be the ones that reported it to SolusVM. When we reported the issue, they basically shrugged.

We try to be super careful around SolusVM, and unfortunately where we're not careful these things slip. I tried avoiding adding templates for a very long time and we got a lot of negative feedback for that decision because we wanted to go through and security harden the templates or create our own, but eventually, we gave in due to time constraints and put out the default templates without thoroughly auditing them. We do try to label our templates, I'm not sure where exactly it does or doesn't get displayed, but this Debian 10 template is listed specifically as "Default SSH port 22. Please note, this is default OS with no security configuration."

We're going to really try to prioritize some template overhauls soon. Debian 10 has been disabled and we're sending out an e-mail. Apologies for the late notice on our part.

rm_

it might be worth contacting the pool operator and letting them know the address so they can freeze future payouts for the user

Oh and please then publish the pool names whose operators who comply with such a request. That is, pools which will "freeze payouts" on an arbitrary request of an unrelated third party -- so that we could avoid such pools.

linwong

I installed Debian 10 on a recently purchased VPS from Virmach back in December. I do not see any evidence of a debianuser on my system.

I read that Virmach should have been affected, and I did install with Debian 10 under SolusVM. I'm just curious as to why I don't have that debianuser on my system when it seems like I should have. Anybody else like this?

entrailz

@rm_ said:

it might be worth contacting the pool operator and letting them know the address so they can freeze future payouts for the user

Oh and please then publish the pool names whose operators who comply with such a request. That is, pools which will "freeze payouts" on an arbitrary request of an unrelated third party -- so that we could avoid such pools.

If you can provide evidence to the pool operator that the address is used with malware, they are likely to investigate and take action. Just one off of an address isn't going to do it.

rm_

If you can provide evidence to the pool operator that the address is used with malware, they are likely to investigate and take action.

Do not imagine yourself to be "the internet police", you are not the police, they are not the police, there are no grounds for them to withhold money from any particular mining address -- and do not confuse this for a webhost which distributes malware (and could take it down when noticed), the relationship and who-does-what or who-enables-what here are all different.

Secondly, why do you believe that it is the pool owner who should get the mined profits? Clearly they can't "return" it to anyone or to those who got the VPSes exploited. So if any pool complies, it's just that "oh I can get some free money, while possibly looking good by 'fighting malware', let's do that".

The crypto-currency embodies a certain ideology as well, and your suggestion to block the address is a total opposite, from the same line of thinking where "Microsoft and Google should unite and disconnect bad domains", all the way to "my government should just block incorrect opinions on the Internet".

Just secure your damn systems.

MikePT

Forgot to update you yesterday guys.

I've reached a Senior there and SolusVM will be notifying all their customers.

alwaysintheprovidersandcustomersside

brueggus

@MikePT said: SolusVM will be notifying all their customers.

It's too late to apologize.
It's too late...
I said it's too late to apologize.
It's too late...

msg7086

@MikePT said:

@msg7086 said:

That's exactly what I thought when I was making our own templates. They don't support new features like uninit_bg and 64 bit, so you have to turn them off if you make your own ext4 partition. I guess they just revert to ext3 and call it a day.

That's a workaround, not a fix. I was able to create a KVM image for Proxmox just fine though!

The reason is that they rely on tools to manipulate the VM partitions to do jobs like changing root passwords or reconfigure networking. However because solusvm is based on RH family 7, the stable version of those tools apparently don't support those new features. So either, like us, to create ext4 partitions with features turned off, or, if installed from an ISO, to use ext3 instead. (Let me guess, they don't know how to install an OS without using the CD Installer.)

Proxmox instead use cloud-init, and doesn't need those crappy magic.

mcgree

@VirMach said:

@Unbelievable said: I'm guessing @virmach is going to have a large drop in server load and same with @hosthatch and racknerd

I don't believe this issue is widespread. While the vulnerability definitely exists, I do not believe many customers were compromised. We have systems in place that basically block, to some low level, anyone going around trying to log into people's servers.

This is something that would definitely have a huge impact if it were widespread, with the hackers running a crypto miner.

@Daniel15 said: I wonder if any LXC or OpenVZ templates have a similar issue...

We did have an issue with people's services on OpenVZ, to some degree, having xmrig, but most of these users when we dove in and handled the case, if not nearly all, had an insecure root password or the customer was deliberately mass mining.

@Daniel15 said: so this has been in the wild for at least four months (probably longer)

This has most likely been around since at least October 2019.

@MikePT said: Has SolusVM emailed the providers about this?

@udonworld said: Where is the announcement by SolusVM?

@Neoon said: Inform anyone that may use SolusVM, surely, enough people will find each other to pull SolusVM to court.
Or even better, boycott SolusVM as costumer.

I don't believe they notified customers, nor would I see them doing that. The last time we had a close call we had to have someone locate it, and then we had to be the ones that reported it to SolusVM. When we reported the issue, they basically shrugged.

We try to be super careful around SolusVM, and unfortunately where we're not careful these things slip. I tried avoiding adding templates for a very long time and we got a lot of negative feedback for that decision because we wanted to go through and security harden the templates or create our own, but eventually, we gave in due to time constraints and put out the default templates without thoroughly auditing them. We do try to label our templates, I'm not sure where exactly it does or doesn't get displayed, but this Debian 10 template is listed specifically as "Default SSH port 22. Please note, this is default OS with no security configuration."

We're going to really try to prioritize some template overhauls soon. Debian 10 has been disabled and we're sending out an e-mail. Apologies for the late notice on our part.

I think Debian 11 will be released soon, and I would like to know when the OS template will be available.

randomq

I backed up the debianuser files and dumped the RAM of the mining and backdoor processes. Got a few IP addresses from lsof and the mem dump ENVs. No Monero private keys, sadly, but this was the address being payed (and still being payed) on https://supportxmr.com/

43hpviAqezAaSKg7g6nTruYUxvdoEbedQQxvAsKEoZbpfWMpwDAbb4tXdPmMYJ1wc4afLFPR5UPVW66Hg6uoXQDf9zyFEYc

They are mining at 34,000 H/s, so maybe a few hundred cores or systems on that particular address. They've made about $15 bucks so far, totally worth it! ^_^

stefeman

Did anyone figure the password out yet? Hopefully it was not "test123" or anything similar..

edit: https://tdn.solusvm.com/ well fuck.. it has been public since 2019.. Time to re-install everything.

AlwaysHappy

@stefeman said:
Did anyone figure the password out yet? Hopefully it was not "test123" or anything similar..

edit: https://tdn.solusvm.com/ well fuck.. it has been public since 2019.. Time to re-install everything.

duckeeyuck

@randomq said: I backed up the debianuser files and dumped the RAM of the mining and backdoor processes. Got a few IP addresses from lsof and the mem dump ENVs. No Monero private keys, sadly, but this was the address being payed (and still being payed) on https://supportxmr.com/

Hey, what country do you get by looking those IPs up?
I recently got 0day'd and had the exact sameuser on my instance, now I'm thinking it might not have been a 0day but this instead, ended up letting it got for a bit trying to find out more of how this worked and who/what could be behind it, it was a long run and educational w/e, but eventually left it.

yoursunny

@stefeman said:
Did anyone figure the password out yet? Hopefully it was not "test123" or anything similar..

edit: https://tdn.solusvm.com/ well fuck.. it has been public since 2019.. Time to re-install everything.

So the debianuser password is same as the root password?
It should be possible to setup a honeypot.

DP

@yoursunny said: It should be possible to setup a honeypot.

It should also be possible to sue Solus

randomq

@duckeeyuck said:
Hey, what country do you get by looking those IPs up?

From environment when miner/backdoor were launched:
SSH_CONNECTION=205.185.125.189 38074 45.132.xx.xxx 22 # hadn't changed port yet

Frantech, WY, USA:

NetRange: 205.185.112.0 - 205.185.127.255
CIDR: 205.185.112.0/20
NetName: PONYNET-03
NetHandle: NET-205-185-112-0-1
Parent: NET205 (NET-205-0-0-0-0)
NetType: Direct Allocation
OriginAS: AS53667
Organization: FranTech Solutions (SYNDI-5)
RegDate: 2010-09-03
Updated: 2012-03-25
Ref: https://rdap.arin.net/registry/ip/205.185.112.0

OrgName: FranTech Solutions
OrgId: SYNDI-5
Address: 1621 Central Ave
City: Cheyenne
StateProv: WY
PostalCode: 82001
Country: US
RegDate: 2010-07-21
Updated: 2017-01-28
Ref: https://rdap.arin.net/registry/entity/SYNDI-5

OrgTechHandle: FDI19-ARIN
OrgTechName: Dias, Francisco
OrgTechPhone: +1-778-977-8246
OrgTechEmail: [email protected]
OrgTechRef: https://rdap.arin.net/registry/entity/FDI19-ARIN

OrgAbuseHandle: FDI19-ARIN
OrgAbuseName: Dias, Francisco
OrgAbusePhone: +1-778-977-8246
OrgAbuseEmail: [email protected]
OrgAbuseRef: https://rdap.arin.net/registry/entity/FDI19-ARIN

There were also these. I'm not on my laptop right now so I can't tell you which were the mining process and which were the backdoor process.

149.202.83.171 OVH France
178.128.242.134 DigitalOcean NL
185.92.222.223 Vultr NL
37.187.95.110 OVH France
91.121.140.167 OVH France
94.23.23.52 OVH France
94.23.247.226 OVH France