VestaCP hit with zeroday exploit [May 19 Security Update]

Harambe

Lots of users on the forum reporting their boxes were hacked. VestaCP team members suggest shutting down the vesta service on your box until they can figure it out and release a patch.

https://forum.vestacp.com/viewtopic.php?f=10&t=16556

Double check your /etc/cron.hourly folder for a file named gcc.sh - you don't want to see that file there.

None of my boxes seem to be impacted, but disable the vesta service:

service vesta stop / systemctl stop vesta

And make sure your admin panel (:8083) isn't loading. Better to be safe than sorry.

April 10 Update: Unclear if patch resolved the exploit. VestaCP team has not produced confirmed details on the attack vector and have not been able to reproduce the attack. Harden your VestaCP installs by keeping the vesta service offline and/or locking down admin ports in firewall.

---

Patch Release!

Patch was just released, hard to tell if this is the final fix though:

https://forum.vestacp.com/viewtopic.php?f=10&t=16556&start=260#p68893

The fix has been released just now!
As usually there are 3 ways to update your server:

1 Via web interface

• Login as admin
• Go to updates tab
• Click un update button under vesta package

2 Via package manager

• SSH as root to your server
• yum update / apt-get update && apt-get upgrade

3 Via GitHub

• SSH as root
• Install git / yum install git /apt-get install git
• Then run following commands

>

cd $(mktemp -d)
git clone git://github.com/serghey-rodin/vesta.git
/bin/cp -rf vesta/* /usr/local/vesta/

Some information about this indecent. We still don't have working exploit for previous version. But we know for sure that the vector of attack was through a potentially unsecure password check method. Therefore we have completely rewrite password auth function. It's bullet proof now!

Please upgrade your servers as soon as possible.

MasonR

Stickying this for the time being. Hosts and users alike, do you part to secure your machines.

Harambe

Tagged on a quick edit. Look for a gcc.sh file in your cron folders, specifically cron.hourly.

Definitely disable the vesta service to cover your ass.

austenite

How long until the 'I'm losing MILLIONS every hour because of Vesta' posts?

YokedEgg

I'm losing MILLIONS every hour because of Vesta.

Mic-hael

Vesta plans, on pause.

teamacc

@austenite said:
How long until the 'I'm losing MILLIONS every hour because of Vesta' posts?

I'm losing MILLIONS every hour because of someone posting "I'm losing MILLIONS every hour because of Vesta"

YokedEgg

I'm losing MILLIONS every hour because of someone posting "I'm losing MILLIONS every hour because of someone posting "I'm losing MILLIONS every hour because of Vesta""

Vova1234

Anti-Hack worked quickly and immediately.

But VPS immediately went to SUSPENSION.

The servers are in rescue-pro.

IP is just blocked by anti-hack.

Dylan

My Vesta box was hit this morning. Not what I wanted to spend my Saturday on but I ended up migrating to a fully self-setup stack... with Vesta's glacial update pace (one single update in the entirety of 2017!) I wouldn't expect a quick fix.

Saragoldfarb

I didn't get hit. Maybe becouse I'm not running on their standard ports. I shutdown Vesta altogther to be sure. Curious to see what they find. Thanks for letting us know.

Setsura

Good thing I don't use any panels :^)

Harambe

@Saragoldfarb said:
I didn't get hit. Maybe becouse I'm not running on their standard ports. I shutdown Vesta altogther to be sure. Curious to see what they find. Thanks for letting us know.

Non-standard ports sounds like a good move in general. One box I have admin & FTP ports locked down to my IPs as well. Disabled the panel completely for good measure though.

Radi

Disabled Vesta service on our shared hosting until this is fixed. Thanks.

Hxxx

Well, what a shame. Hopefully nobody leaked anything important with this zero day exploit. Stick to cPanel or straight vanilla/console.

warrior

some of my friends using vestacp because easiest to manage, i'll notice it. thanks

Saragoldfarb

@Dylan said:
My Vesta box was hit this morning. Not what I wanted to spend my Saturday on but I ended up migrating to a fully self-setup stack... with Vesta's glacial update pace (one single update in the entirety of 2017!) I wouldn't expect a quick fix.

They'll probably put some effort in fixing this bug in a reasonable timeframe. If they don't I'll move away from them as well. Free or not, there's no use in having panel when the authors themselves advise you to disable it

Harambe

Update from one of the VestaCP team members: https://forum.vestacp.com/viewtopic.php?f=10&t=16556&start=60#p68594

Here is what we know so far:
1. The first wave happened on April 4. Servers were infected with /etc/cron.hourly/gcc.sh
2. It was an automated hack
3. CentOS, Debian, Ubuntu all distros are affected it's platform independent
4. We didn't find any traces in vesta and system logs yet
5. On April 7 infected servers started to DDoS remote hosts using /usr/lib/libudev.so.

What you can do:
The best way to stay safe is to temporary disable vesta web service

service vesta stop

systemctl disable vesta

or limit access to port 8083 using firewall

What we are doing:
Few users provided us with root access to their servers. We are investigating what happened. We also launched a couple honeypots in order to get full picture of the hack.

Falzo

what a nightmare

PS @harambe: can't tell you how grateful I am for you posting this here, helped me contain it quickly a lot!!

I am about investigating from my end as much as possible as I had two infected boxes too. will write an update once I find more information (if any)

if you see the gcc.sh note the timestamp and check for files with the same timestamp or changed from then.
the binary also might be found in /lib/libudev.so instead of /usr/lib/libudev.so
it seems like it takes a while for the hack to spread into the system. on a second VM I also found modified /etc/crontab and a file in /etc/init.d and /usr/bin , which were not there for the former VM. so make sure to check closely.

still can't tell how they got in, but from the looks of it, it has to be the separate vesta-service (nginx/php-fpm) itself, maybe an API call?

scorcher9

I was also using it because it is was easy to manage.

Just basic setup, no real site and I still lost millions.

Anyway @Harambe thanks for posting this.

angstrom

It often seems that people choose cPanel if they want to pay, otherwise they choose Vesta, often saying that it's not so great (by the way, does Vesta support IPv6 yet?). Just curious: why not choose one of the other (good) free panels instead of Vesta, such as Froxlor or Webmin/Virtualmin or CentOS Web Panel?

(I don't have a lot of experience with different panels.)

jetchirag

angstrom said: Vesta

I'd say it comes down to preferences. Vestacp offers easy installation/customisation/configuration and security issues can be found in any panel. Many people do use other panels especially virtualmin.

Harambe

@Falzo said:
what a nightmare

PS @harambe: can't tell you how grateful I am for you posting this here, helped me contain it quickly a lot!!

No worries man. Figured more than a few folks here also use VestaCP, don't want anyone getting pwned if it can be prevented.

Also: to anyone who was infected, please consider joining the vesta forum and helping the devs get to the bottom of this. They've had a couple releases in the past few months, which is a nice change after a year w/o a release, and seem keen on getting this fixed.

So if you have any info to contribute or can give them access to a pwned install, please consider sharing it directly.

Harambe

@austenite said:
How long until the 'I'm losing MILLIONS every hour because of Vesta' posts?

Found 'em, on the VestaCP forum thread

scorcher9

angstrom said: why not choose one of the other (good) free panels

What would be the criteria for choosing one?

Anyone can do a simple search for vulnerabilities and land on something like this:

Vesta:
https://www.cvedetails.com/vulnerability-list/vendor_id-15494/product_id-31935/Vestacp-Vesta-Control-Panel.html

Froxlor:
https://www.cvedetails.com/vulnerability-list/vendor_id-16113/Froxlor.html

Webmin:
https://www.cvedetails.com/vulnerability-list/vendor_id-358/Webmin.html

Webmin has the most. So how do you decide then?

angstrom

@scorcher9 said:

angstrom said: why not choose one of the other (good) free panels

What would be the criteria for choosing one?

Anyone can do a simple search for vulnerabilities and land on something like this:

Vesta:
https://www.cvedetails.com/vulnerability-list/vendor_id-15494/product_id-31935/Vestacp-Vesta-Control-Panel.html

Froxlor:
https://www.cvedetails.com/vulnerability-list/vendor_id-16113/Froxlor.html

Webmin:
https://www.cvedetails.com/vulnerability-list/vendor_id-358/Webmin.html

Webmin has the most. So how do you decide then?

Okay, fine, if those pages are taken as definitive, then Vesta has only ever had one security vulnerability until now, so I guess that on that basis, I would choose it as well.

angstrom

This from two years ago:

https://stackoverflow.com/questions/36623596/is-this-file-gcc-sh-in-cron-hourly-malware

I wonder whether it's related.

Harambe

@angstrom said:

I wonder whether it's related.

Yep, they're using a variant of Xor DDoS - https://en.wikipedia.org/wiki/Xor_DDoS

scorcher9

angstrom said: Okay, fine, if those pages are taken as definitive, then Vesta has only ever had one security vulnerability until now, so I guess that on that basis, I would choose it as well.

Not sure about taking them as definitive but I was just asking how to know which panel is good and which is not?

HBAndrei

Only have one VestaCP box and its 8083 port is closed off in the firewall... but this is somewhat concerning, so I stopped the entire vesta service as recommended. Thanks for sharing.

angstrom

@scorcher9 said:

angstrom said: Okay, fine, if those pages are taken as definitive, then Vesta has only ever had one security vulnerability until now, so I guess that on that basis, I would choose it as well.

Not sure about taking them as definitive but I was just asking how to know which panel is good and which is not?

Of course, I wasn't entirely serious about "definitive", but the number of (discovered) security vulnerabilities could be used as a criterion for choosing between the free panels (why not?). :-) This said, I find it hard to believe that Vesta has only had one security vulnerability until now ...

scorcher9

angstrom said: I find it hard to believe that Vesta has only had one security vulnerability until now

True

doughmanes

Wait, wasn't the community giving a certain guy a bunch of shit over his claims that VESTACP IS GARBAGE AND BELONGS IN THE TRASH

TOLD YOU SO - TOLD YOU SO - TOLD YOU SO - TOLD YOU SO - TOLD YOU SO - TOLD YOU SO

PremiumN

@doughmanes said:
Wait, wasn't the community giving a certain guy a bunch of shit over his claims that VESTACP IS GARBAGE AND BELONGS IN THE TRASH

I'd tell that guy to go and built something better and show.

Just because there is a security exploit doesn't mean that the product is trash.

Prime404

It seems like a forum member on the VestaCP forums may have discovered what's being used to exploit the servers..
https://forum.vestacp.com/viewtopic.php?f=10&t=16556&start=170#p68739

Hxxx

Is trash, you are better off using a better trash like cPanel.

@PremiumN said:

@doughmanes said:
Wait, wasn't the community giving a certain guy a bunch of shit over his claims that VESTACP IS GARBAGE AND BELONGS IN THE TRASH

I'd tell that guy to go and built something better and show.

Just because there is a security exploit doesn't mean that the product is trash.

squibs

My nodes are all clean thankfully. Vesta shut down for now

FHR

I like Plesk, actually a usable hosting panel. It's expensive though

FlamesRunner

Great, now I'm curious of what's in gcc.sh.

Saragoldfarb

@Prime404 said:
It seems like a forum member on the VestaCP forums may have discovered what's being used to exploit the servers..
https://forum.vestacp.com/viewtopic.php?f=10&t=16556&start=170#p68739

Confirmed by Vesta as far as I can tell. They say a patch is coming today.

jar

@doughmanes said:
Wait, wasn't the community giving a certain guy a bunch of shit over his claims that VESTACP IS GARBAGE AND BELONGS IN THE TRASH

TOLD YOU SO - TOLD YOU SO - TOLD YOU SO - TOLD YOU SO - TOLD YOU SO - TOLD YOU SO

Yes I gave that crowd a ton of shit. If I threw out everything imperfect I'd be naked in the corner of an empty room talking about how great I am, which those people are one logical step away from.

doughmanes

I think the Kloxo fans jumped headfirst into the VestaCP ship

Prime404

@Saragoldfarb said:

@Prime404 said:
It seems like a forum member on the VestaCP forums may have discovered what's being used to exploit the servers..
https://forum.vestacp.com/viewtopic.php?f=10&t=16556&start=170#p68739

Confirmed by Vesta as far as I can tell. They say a patch is coming today.

Yeah, by studying the code.. it seems to be what's at fault here. Wouldn't surprise me if we see more of these attacks as some of the code for like the API seems to be very unsafe in certain ways.

MikePT

I would recommend the providers to issue security advisors as this may have a huge impact. There is many people using VestaCP. This doesn't seem to start only DDoS but SPAM as well, anything actually would be possible since it runs as root so you should definitely notify your clients as this causes major issues to both parties. If someone here would be willing to pass me the details for a compromised VM so I can investigate this further and narrow the root of the issue I would appreciate. From the comments I have been reading this may be a vulnerability in the API. Roundcube should be excluded for now. I do not have any VestaCP servers as I no longer consider those secure enough. Their team had plenty of time to address this potential security issue. Looking at their changelogs I don't think they take it serious. I know its a FOSS project and I am thankfull for contributing to the OpenSource community, although the project itself is no longer secure and their team's attittude towards this matter is ridiculous. May actually cause more harm than good.

Janevski

@scorcher9 said:

angstrom said: why not choose one of the other (good) free panels

What would be the criteria for choosing one?

Anyone can do a simple search for vulnerabilities and land on something like this:

Vesta:
https://www.cvedetails.com/vulnerability-list/vendor_id-15494/product_id-31935/Vestacp-Vesta-Control-Panel.html

Froxlor:
https://www.cvedetails.com/vulnerability-list/vendor_id-16113/Froxlor.html

Webmin:
https://www.cvedetails.com/vulnerability-list/vendor_id-358/Webmin.html

Webmin has the most. So how do you decide then?

Look for which one has the most shameful exploits and avoid those slackers.

jar

the project itself is no longer secure

Yeah, if it's ever had a code vulnerability you should throw it out and never use it.

Challenge accepted or is that actually not a good idea? Might that leave you only running things you've coded and even then only until you make a mistake, at which point you curl up in the corner and never touch the internet again because everything is so unsafe?

doughmanes

As somebody who worked through Kloxo/NTP amplification attack related security issues in the past which involved working non-stop for a few days, yes opening up tickets to all your customers about issues like this make you look really good to the customer.

I'm not saying start blasting your customers with every vulnerability notification on everything but ones that may impact your service/network from customers being vulnerable.

Encouraging your customers to be on Twitter/FB for alerts like this cut down on email volume

MikePT

@jarland said:

the project itself is no longer secure

Yeah, if it's ever had a code vulnerability you should throw it out and never use it.

Challenge accepted or is that actually not a good idea? Might that leave you only running things you've coded and even then only until you make a mistake, at which point you curl up in the corner and never touch the internet again because everything is so unsafe?

I should rephrase it to:

"I do not think the project was ever secure".

This brings memories from HyperVM / Kloxo.
@jarland please understand that what I mean relates to the developers attittude towards this major flaw and the outdated VestaCP code.

Tom

MikePT said: "I do not think the project was ever secure"

How come?

jar

@MikePT said:

@jarland said:

the project itself is no longer secure

Yeah, if it's ever had a code vulnerability you should throw it out and never use it.

Challenge accepted or is that actually not a good idea? Might that leave you only running things you've coded and even then only until you make a mistake, at which point you curl up in the corner and never touch the internet again because everything is so unsafe?

I should rephrase it to:

"I do not think the project was ever secure".

Late 2013. I've had a pretty good run. Legacy customers knowing that I don't have as much confidence in long term security as cPanel, but it's still a pretty damn good run. In that time there has been a total of 2 concerns.

Prior to that whmcs had repeat concerns within a 1-2 year time frame and we're all still using it.

Just saying the whole "well this had a vulnerability, let's all move to the next one that hasn't yet" isn't a healthy attitude. I'd rather be with the dev who learned from a mistake than the one who hasn't yet. Wait until centoswebpanel gets popular enough... No one who codes flawlessly includes an "install teamspeak" button.

MikePT

Furthermore, I read above someone comparing Webmin to VestaCP in terms of security flaws. There is no comparisson. Webmin is surely used by many millions of people so its obvious they are more often a target. This time it happened to be VestaCP. And gosh if that really correlates to API running as root then... Definitely start using something else.