Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In with OpenID
Advertise on LowEndTalk.com

In this Discussion

VestaCP hit with zeroday exploit [Patch Released, Unclear If Resolved]

VestaCP hit with zeroday exploit [Patch Released, Unclear If Resolved]

HarambeHarambe Member
edited April 10 in General

Lots of users on the forum reporting their boxes were hacked. VestaCP team members suggest shutting down the vesta service on your box until they can figure it out and release a patch.

https://forum.vestacp.com/viewtopic.php?f=10&t=16556

Double check your /etc/cron.hourly folder for a file named gcc.sh - you don't want to see that file there.

None of my boxes seem to be impacted, but disable the vesta service:

service vesta stop / systemctl stop vesta

And make sure your admin panel (:8083) isn't loading. Better to be safe than sorry.

April 10 Update: Unclear if patch resolved the exploit. VestaCP team has not produced confirmed details on the attack vector and have not been able to reproduce the attack. Harden your VestaCP installs by keeping the vesta service offline and/or locking down admin ports in firewall.


Patch Release!

Patch was just released, hard to tell if this is the final fix though:

https://forum.vestacp.com/viewtopic.php?f=10&t=16556&start=260#p68893

The fix has been released just now!
As usually there are 3 ways to update your server:

1 Via web interface
- Login as admin
- Go to updates tab
- Click un update button under vesta package

2 Via package manager
- SSH as root to your server
- yum update / apt-get update && apt-get upgrade

3 Via GitHub
- SSH as root
- Install git / yum install git /apt-get install git
- Then run following commands

cd $(mktemp -d)
git clone git://github.com/serghey-rodin/vesta.git
/bin/cp -rf vesta/* /usr/local/vesta/

Some information about this indecent. We still don't have working exploit for previous version. But we know for sure that the vector of attack was through a potentially unsecure password check method. Therefore we have completely rewrite password auth function. It's bullet proof now!

Please upgrade your servers as soon as possible.

Professional Shoeminer

«13456

Comments

  • MasonRMasonR Moderator

    Stickying this for the time being. Hosts and users alike, do you part to secure your machines.

  • HarambeHarambe Member

    Tagged on a quick edit. Look for a gcc.sh file in your cron folders, specifically cron.hourly.

    Definitely disable the vesta service to cover your ass.

    Professional Shoeminer

  • austeniteaustenite Member, Provider

    How long until the 'I'm losing MILLIONS every hour because of Vesta' posts?

  • YokedEggYokedEgg Member

    I'm losing MILLIONS every hour because of Vesta.

    Yoked or Yolked? You decide.

  • HoostHoost Member, Provider

    Vesta plans, on pause.

    Michael From Ho-ost | Social at HoostSolutions

  • teamaccteamacc Moderator

    @austenite said: How long until the 'I'm losing MILLIONS every hour because of Vesta' posts?

    I'm losing MILLIONS every hour because of someone posting "I'm losing MILLIONS every hour because of Vesta"

    Yo mama so fat each of her butt-cheeks has its own /8.

    Thanked by 1austenite
  • YokedEggYokedEgg Member

    I'm losing MILLIONS every hour because of someone posting "I'm losing MILLIONS every hour because of someone posting "I'm losing MILLIONS every hour because of Vesta""

    Yoked or Yolked? You decide.

  • Vova1234Vova1234 Member, Provider
    edited April 8

    Anti-Hack worked quickly and immediately.

    But VPS immediately went to SUSPENSION.

    The servers are in rescue-pro.

    IP is just blocked by anti-hack.

    My Hosting: ABCVG.net

  • DylanDylan Member
    edited April 8

    My Vesta box was hit this morning. Not what I wanted to spend my Saturday on but I ended up migrating to a fully self-setup stack... with Vesta's glacial update pace (one single update in the entirety of 2017!) I wouldn't expect a quick fix.

  • SaragoldfarbSaragoldfarb Member
    edited April 8

    I didn't get hit. Maybe becouse I'm not running on their standard ports. I shutdown Vesta altogther to be sure. Curious to see what they find. Thanks for letting us know.

  • SetsuraSetsura Member

    Good thing I don't use any panels :^)

    Will shill for Pop-Tarts(must be strawberry flavour).

    Thanked by 1ariq01
  • HarambeHarambe Member

    @Saragoldfarb said: I didn't get hit. Maybe becouse I'm not running on their standard ports. I shutdown Vesta altogther to be sure. Curious to see what they find. Thanks for letting us know.

    Non-standard ports sounds like a good move in general. One box I have admin & FTP ports locked down to my IPs as well. Disabled the panel completely for good measure though.

    Professional Shoeminer

    Thanked by 1Saragoldfarb
  • RadiRadi Member, Provider

    Disabled Vesta service on our shared hosting until this is fixed. Thanks.

    VikingLayer now offers VPS resource pools. Ask me about them today. :)

  • HxxxHxxx Member

    Well, what a shame. Hopefully nobody leaked anything important with this zero day exploit. Stick to cPanel or straight vanilla/console.

  • warriorwarrior Member

    some of my friends using vestacp because easiest to manage, i'll notice it. thanks

  • @Dylan said: My Vesta box was hit this morning. Not what I wanted to spend my Saturday on but I ended up migrating to a fully self-setup stack... with Vesta's glacial update pace (one single update in the entirety of 2017!) I wouldn't expect a quick fix.

    They'll probably put some effort in fixing this bug in a reasonable timeframe. If they don't I'll move away from them as well. Free or not, there's no use in having panel when the authors themselves advise you to disable it :)

  • HarambeHarambe Member
    edited April 8

    Update from one of the VestaCP team members: https://forum.vestacp.com/viewtopic.php?f=10&t=16556&start=60#p68594

    Here is what we know so far:
    1. The first wave happened on April 4. Servers were infected with /etc/cron.hourly/gcc.sh
    2. It was an automated hack
    3. CentOS, Debian, Ubuntu all distros are affected it's platform independent
    4. We didn't find any traces in vesta and system logs yet
    5. On April 7 infected servers started to DDoS remote hosts using /usr/lib/libudev.so.

    What you can do:
    The best way to stay safe is to temporary disable vesta web service

    service vesta stop

    systemctl disable vesta

    or limit access to port 8083 using firewall

    What we are doing:
    Few users provided us with root access to their servers. We are investigating what happened. We also launched a couple honeypots in order to get full picture of the hack.

    Professional Shoeminer

  • FalzoFalzo Member
    edited April 8

    what a nightmare :(

    PS @harambe: can't tell you how grateful I am for you posting this here, helped me contain it quickly a lot!!

    I am about investigating from my end as much as possible as I had two infected boxes too. will write an update once I find more information (if any)

    if you see the gcc.sh note the timestamp and check for files with the same timestamp or changed from then.
    the binary also might be found in /lib/libudev.so instead of /usr/lib/libudev.so
    it seems like it takes a while for the hack to spread into the system. on a second VM I also found modified /etc/crontab and a file in /etc/init.d and /usr/bin , which were not there for the former VM. so make sure to check closely.

    still can't tell how they got in, but from the looks of it, it has to be the separate vesta-service (nginx/php-fpm) itself, maybe an API call?

    UltraVPS.eu KVM in US/UK/NL/DE: 15% off first 6 month and cheap 750G / 2TB storage offers
    Netcup KVM: 2GB 40TB BW - 16,14€ 6m or 2 dedCore 6GB 320GB - 78,88€ 12m /w 5€ off: 36nc15222153920 - 36nc15222153929

    Thanked by 2mehargags Ympker
  • scorcher9scorcher9 Member
    edited April 8

    I was also using it because it is was easy to manage.

    Just basic setup, no real site and I still lost millions.

    Anyway @Harambe thanks for posting this.

  • angstromangstrom Member

    It often seems that people choose cPanel if they want to pay, otherwise they choose Vesta, often saying that it's not so great (by the way, does Vesta support IPv6 yet?). Just curious: why not choose one of the other (good) free panels instead of Vesta, such as Froxlor or Webmin/Virtualmin or CentOS Web Panel?

    (I don't have a lot of experience with different panels.)

    "[T]he number of UNIX installations has grown to 16, with more expected." (K. Thompson & D. M. Ritchie, UNIX Programmer's Manual, 3ed, 1973)

    Thanked by 3Saragoldfarb v3ng lazyt
  • angstrom said: Vesta

    I'd say it comes down to preferences. Vestacp offers easy installation/customisation/configuration and security issues can be found in any panel. Many people do use other panels especially virtualmin.

    Doo-doo-doo, doo-doo-doo

    Doo-doo-doo, doo-doo-doo

    Thanked by 1angstrom
  • HarambeHarambe Member
    edited April 8

    @Falzo said: what a nightmare :(

    PS @harambe: can't tell you how grateful I am for you posting this here, helped me contain it quickly a lot!!

    No worries man. Figured more than a few folks here also use VestaCP, don't want anyone getting pwned if it can be prevented.

    Also: to anyone who was infected, please consider joining the vesta forum and helping the devs get to the bottom of this. They've had a couple releases in the past few months, which is a nice change after a year w/o a release, and seem keen on getting this fixed.

    So if you have any info to contribute or can give them access to a pwned install, please consider sharing it directly.

    Professional Shoeminer

    Thanked by 2MasonR mehargags
  • HarambeHarambe Member

    @austenite said: How long until the 'I'm losing MILLIONS every hour because of Vesta' posts?

    Found 'em, on the VestaCP forum thread

    Professional Shoeminer

  • angstrom said: why not choose one of the other (good) free panels

    What would be the criteria for choosing one?

    Anyone can do a simple search for vulnerabilities and land on something like this:

    Vesta:
    https://www.cvedetails.com/vulnerability-list/vendor_id-15494/product_id-31935/Vestacp-Vesta-Control-Panel.html
    
    Froxlor:
    https://www.cvedetails.com/vulnerability-list/vendor_id-16113/Froxlor.html
    
    Webmin:
    https://www.cvedetails.com/vulnerability-list/vendor_id-358/Webmin.html
    

    Webmin has the most. So how do you decide then?

    Thanked by 1angstrom
  • angstromangstrom Member

    @scorcher9 said:

    angstrom said: why not choose one of the other (good) free panels

    What would be the criteria for choosing one?

    Anyone can do a simple search for vulnerabilities and land on something like this:

    Vesta:
    https://www.cvedetails.com/vulnerability-list/vendor_id-15494/product_id-31935/Vestacp-Vesta-Control-Panel.html
    
    Froxlor:
    https://www.cvedetails.com/vulnerability-list/vendor_id-16113/Froxlor.html
    
    Webmin:
    https://www.cvedetails.com/vulnerability-list/vendor_id-358/Webmin.html
    

    Webmin has the most. So how do you decide then?

    Okay, fine, if those pages are taken as definitive, then Vesta has only ever had one security vulnerability until now, so I guess that on that basis, I would choose it as well.

    "[T]he number of UNIX installations has grown to 16, with more expected." (K. Thompson & D. M. Ritchie, UNIX Programmer's Manual, 3ed, 1973)

    Thanked by 1scorcher9
  • angstromangstrom Member

    This from two years ago:

    https://stackoverflow.com/questions/36623596/is-this-file-gcc-sh-in-cron-hourly-malware

    I wonder whether it's related.

    "[T]he number of UNIX installations has grown to 16, with more expected." (K. Thompson & D. M. Ritchie, UNIX Programmer's Manual, 3ed, 1973)

  • HarambeHarambe Member

    @angstrom said:

    I wonder whether it's related.

    Yep, they're using a variant of Xor DDoS - https://en.wikipedia.org/wiki/Xor_DDoS

    Professional Shoeminer

    Thanked by 1angstrom
  • angstrom said: Okay, fine, if those pages are taken as definitive, then Vesta has only ever had one security vulnerability until now, so I guess that on that basis, I would choose it as well.

    Not sure about taking them as definitive but I was just asking how to know which panel is good and which is not?

  • HBAndreiHBAndrei Member, Provider

    Only have one VestaCP box and its 8083 port is closed off in the firewall... but this is somewhat concerning, so I stopped the entire vesta service as recommended. Thanks for sharing.

    Free Uptime Monitoring - minimize your downtime by being the first to know about it.
    Free Blacklist Monitoring - don't let a few bad clients ruin your network.

  • angstromangstrom Member

    @scorcher9 said:

    angstrom said: Okay, fine, if those pages are taken as definitive, then Vesta has only ever had one security vulnerability until now, so I guess that on that basis, I would choose it as well.

    Not sure about taking them as definitive but I was just asking how to know which panel is good and which is not?

    Of course, I wasn't entirely serious about "definitive", but the number of (discovered) security vulnerabilities could be used as a criterion for choosing between the free panels (why not?). :-) This said, I find it hard to believe that Vesta has only had one security vulnerability until now ...

    "[T]he number of UNIX installations has grown to 16, with more expected." (K. Thompson & D. M. Ritchie, UNIX Programmer's Manual, 3ed, 1973)

  • angstrom said: I find it hard to believe that Vesta has only had one security vulnerability until now

    True :)

  • doughmanesdoughmanes Member
    edited April 8

    Wait, wasn't the community giving a certain guy a bunch of shit over his claims that VESTACP IS GARBAGE AND BELONGS IN THE TRASH

    TOLD YOU SO - TOLD YOU SO - TOLD YOU SO - TOLD YOU SO - TOLD YOU SO - TOLD YOU SO

    I'm like your older brother/uncle who makes fun of you, you hate it and once you let your emotions run their course you start to realize my point.

    Thanked by 1vimalware
  • PremiumNPremiumN Member

    @doughmanes said: Wait, wasn't the community giving a certain guy a bunch of shit over his claims that VESTACP IS GARBAGE AND BELONGS IN THE TRASH

    I'd tell that guy to go and built something better and show.

    Just because there is a security exploit doesn't mean that the product is trash.

    I ❤ Laravel

  • Prime404Prime404 Member

    It seems like a forum member on the VestaCP forums may have discovered what's being used to exploit the servers.. https://forum.vestacp.com/viewtopic.php?f=10&t=16556&start=170#p68739

    Thanked by 1FHR
  • HxxxHxxx Member

    Is trash, you are better off using a better trash like cPanel.

    @PremiumN said:

    @doughmanes said: Wait, wasn't the community giving a certain guy a bunch of shit over his claims that VESTACP IS GARBAGE AND BELONGS IN THE TRASH

    I'd tell that guy to go and built something better and show.

    Just because there is a security exploit doesn't mean that the product is trash.

    Thanked by 1doughmanes
  • squibssquibs Member

    My nodes are all clean thankfully. Vesta shut down for now

    ¦̵̱ ̵̱ ̵̱ ̵̱ ̵̱(̢ ̡͇̅└͇̅┘͇̅ (▤8כ−◦

  • FHRFHR Member

    I like Plesk, actually a usable hosting panel. It's expensive though

  • Great, now I'm curious of what's in gcc.sh.

    Benchmark: wget https://s.flamz.pw/dl/bench.sh && bash bench.sh

  • @Prime404 said: It seems like a forum member on the VestaCP forums may have discovered what's being used to exploit the servers.. https://forum.vestacp.com/viewtopic.php?f=10&t=16556&start=170#p68739

    Confirmed by Vesta as far as I can tell. They say a patch is coming today.

  • jarlandjarland Provider

    @doughmanes said: Wait, wasn't the community giving a certain guy a bunch of shit over his claims that VESTACP IS GARBAGE AND BELONGS IN THE TRASH

    TOLD YOU SO - TOLD YOU SO - TOLD YOU SO - TOLD YOU SO - TOLD YOU SO - TOLD YOU SO

    Yes I gave that crowd a ton of shit. If I threw out everything imperfect I'd be naked in the corner of an empty room talking about how great I am, which those people are one logical step away from.

    Thanked by 1jvnadr
  • I think the Kloxo fans jumped headfirst into the VestaCP ship

    I'm like your older brother/uncle who makes fun of you, you hate it and once you let your emotions run their course you start to realize my point.

    Thanked by 2mehargags vimalware
  • Prime404Prime404 Member

    @Saragoldfarb said:

    @Prime404 said: It seems like a forum member on the VestaCP forums may have discovered what's being used to exploit the servers.. https://forum.vestacp.com/viewtopic.php?f=10&t=16556&start=170#p68739

    Confirmed by Vesta as far as I can tell. They say a patch is coming today.

    Yeah, by studying the code.. it seems to be what's at fault here. Wouldn't surprise me if we see more of these attacks as some of the code for like the API seems to be very unsafe in certain ways.

  • MikePTMikePT Member, Provider
    edited April 8

    I would recommend the providers to issue security advisors as this may have a huge impact. There is many people using VestaCP. This doesn't seem to start only DDoS but SPAM as well, anything actually would be possible since it runs as root so you should definitely notify your clients as this causes major issues to both parties. If someone here would be willing to pass me the details for a compromised VM so I can investigate this further and narrow the root of the issue I would appreciate. From the comments I have been reading this may be a vulnerability in the API. Roundcube should be excluded for now. I do not have any VestaCP servers as I no longer consider those secure enough. Their team had plenty of time to address this potential security issue. Looking at their changelogs I don't think they take it serious. I know its a FOSS project and I am thankfull for contributing to the OpenSource community, although the project itself is no longer secure and their team's attittude towards this matter is ridiculous. May actually cause more harm than good.

    MXroute.io - SMTP Relay Service, powered by MailChannels, fully automated, LET plans

    MXroute.com - Email Hosting, powered by MailChannels

    Thanked by 1doughmanes
  • JanevskiJanevski Member

    @scorcher9 said:

    angstrom said: why not choose one of the other (good) free panels

    What would be the criteria for choosing one?

    Anyone can do a simple search for vulnerabilities and land on something like this:

    Vesta:
    https://www.cvedetails.com/vulnerability-list/vendor_id-15494/product_id-31935/Vestacp-Vesta-Control-Panel.html
    
    Froxlor:
    https://www.cvedetails.com/vulnerability-list/vendor_id-16113/Froxlor.html
    
    Webmin:
    https://www.cvedetails.com/vulnerability-list/vendor_id-358/Webmin.html
    

    Webmin has the most. So how do you decide then?

    Look for which one has the most shameful exploits and avoid those slackers.

  • jarlandjarland Provider
    edited April 8

    the project itself is no longer secure

    Yeah, if it's ever had a code vulnerability you should throw it out and never use it.

    Challenge accepted or is that actually not a good idea? Might that leave you only running things you've coded and even then only until you make a mistake, at which point you curl up in the corner and never touch the internet again because everything is so unsafe? :)

  • doughmanesdoughmanes Member
    edited April 8

    As somebody who worked through Kloxo/NTP amplification attack related security issues in the past which involved working non-stop for a few days, yes opening up tickets to all your customers about issues like this make you look really good to the customer.

    I'm not saying start blasting your customers with every vulnerability notification on everything but ones that may impact your service/network from customers being vulnerable.

    Encouraging your customers to be on Twitter/FB for alerts like this cut down on email volume

    I'm like your older brother/uncle who makes fun of you, you hate it and once you let your emotions run their course you start to realize my point.

    Thanked by 1MikePT
  • MikePTMikePT Member, Provider
    edited April 8

    @jarland said:

    the project itself is no longer secure

    Yeah, if it's ever had a code vulnerability you should throw it out and never use it.

    Challenge accepted or is that actually not a good idea? Might that leave you only running things you've coded and even then only until you make a mistake, at which point you curl up in the corner and never touch the internet again because everything is so unsafe? :)

    I should rephrase it to:

    "I do not think the project was ever secure".

    This brings memories from HyperVM / Kloxo. @jarland please understand that what I mean relates to the developers attittude towards this major flaw and the outdated VestaCP code.

    MXroute.io - SMTP Relay Service, powered by MailChannels, fully automated, LET plans

    MXroute.com - Email Hosting, powered by MailChannels

  • TomTom Member

    MikePT said: "I do not think the project was ever secure"

    How come?

  • jarlandjarland Provider
    edited April 8

    @MikePT said:

    @jarland said:

    the project itself is no longer secure

    Yeah, if it's ever had a code vulnerability you should throw it out and never use it.

    Challenge accepted or is that actually not a good idea? Might that leave you only running things you've coded and even then only until you make a mistake, at which point you curl up in the corner and never touch the internet again because everything is so unsafe? :)

    I should rephrase it to:

    "I do not think the project was ever secure".

    Late 2013. I've had a pretty good run. Legacy customers knowing that I don't have as much confidence in long term security as cPanel, but it's still a pretty damn good run. In that time there has been a total of 2 concerns.

    Prior to that whmcs had repeat concerns within a 1-2 year time frame and we're all still using it.

    Just saying the whole "well this had a vulnerability, let's all move to the next one that hasn't yet" isn't a healthy attitude. I'd rather be with the dev who learned from a mistake than the one who hasn't yet. Wait until centoswebpanel gets popular enough... No one who codes flawlessly includes an "install teamspeak" button.

    Thanked by 2Plioser coreflux
  • MikePTMikePT Member, Provider

    Furthermore, I read above someone comparing Webmin to VestaCP in terms of security flaws. There is no comparisson. Webmin is surely used by many millions of people so its obvious they are more often a target. This time it happened to be VestaCP. And gosh if that really correlates to API running as root then... Definitely start using something else.

    MXroute.io - SMTP Relay Service, powered by MailChannels, fully automated, LET plans

    MXroute.com - Email Hosting, powered by MailChannels

Sign In or Register to comment.