Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In with OpenID
Sign In Register
Advertise on LowEndTalk.com

Categories

In this Discussion

› › ChicagoVPS hacked, bunch of VPS customers offline

ChicagoVPS hacked, bunch of VPS customers offline

pubcrawlerpubcrawler Banned
edited November 2012 in General

Got an email 2+ hours ago directly from ChicagoVPS (am a customer):

[CRITICAL UPDATE]

re: Chicago VPS11, Chicago VPS12, Chicago VPS14, Chicago VPS16, Chicago VPS17, Chicago VPS26, Chicago VPS28, Chicago VPS29, Chicago VPS30, Chicago VPS31

ChicagoVPS experienced a brute force on the SolusVM API for the administrative section. This caused the above affected nodes to become compromised before we were able to stop the attack.

What does this mean? Currently the VM's on these nodes are being recovered to the fullest ability of Chicago VPS staff from the incomplete data destruction process and from central backups. Any VM's unable to be recreated from the remaining data or from backups will be created fresh.

ChicagoVPS is committed to customer satisfaction and any way in our ability will do what we can to get everyone back up and going as fast and as best as we can.

We will post additional updates on twitter and facebook and from time to time send out an email regarding the current status of the progress.

If you have any questions in the mean time, feel free to directly email me at jshinkle@chicagovps.net

Sincerely,

Jeremiah L. Shinkle Chief Networking Officer ChicagoVPS

@CVPS_Chris loves --> http://www.hawkhost.com/

Tagged:
«134567…10

Comments

  • ShardHostSarahShardHostSarah Member

    Thats a pretty crappy thing to happen for both client and provider. Is this an exploit in SolusVM or something not locked down correctly?

    AnnualKVM KVM VPS from £10/year

  • TazTaz Disabled

    Karma is a bitch. Hope it is not too bad and they can recover.

    Time is good and also bad. Life is short and that is sad. Dont worry be happy thats my style. No matter what happens i won't lose my smile!

  • jarlandjarland Administrator

    Anyone have a working theory that immediately comes to mind as to what happened here? Admin API would be the API used to connect billing software would it not? Is it not restricted by IP?

    I'm using A2Hosting right now for shared, love it. Referral link.

  • pubcrawlerpubcrawler Banned

    Karma aside, I am wondering where the exploit is and if it's a SolusVM issue. An exploit in SolusVM could impact tons of folks.

    @CVPS_Chris loves --> http://www.hawkhost.com/

  • ihatetonyyihatetonyy Member

    @Taz said: Karma is a bitch.

    Even if their.. marketing, I guess, ruffled some feathers, I don't think it deserves one node getting nearly-trashed, let alone ten.

    With that said, still up over in LA.

    "We are in a prison drama. This is like The Shawshank Redemption, only with more tunneling through shit and no fucking redemption."
  • TazTaz Disabled

    @Liam @infinity please remove /hide ths thread. If this is a solusvm exploit, this can have hugee affect.

    Time is good and also bad. Life is short and that is sad. Dont worry be happy thats my style. No matter what happens i won't lose my smile!

  • jarlandjarland Administrator

    @Taz Nope. If there's an exploit and someone is targeting LEB providers this is the place it should be exposed.

    I'm using A2Hosting right now for shared, love it. Referral link.

  • TazTaz Disabled

    But before solus releases a patch, you are welcoming more skiddies.

    Time is good and also bad. Life is short and that is sad. Dont worry be happy thats my style. No matter what happens i won't lose my smile!

  • CVPS_ChrisCVPS_Chris Member

    @Taz said: But before solus releases a patch, you are welcoming more skiddies.

    Your problem, should probably go deal with that.

    ColoCrossing.com - Premier Datacenter and Cloud Solutions for Business

  • jarlandjarland Administrator
    edited November 2012

    @Taz said: But before solus releases a patch, you are welcoming more skiddies.

    Nope, you're warning LEB providers to watch their butts. Otherwise you're keeping the info from them to let them get targeted if this is going to continue through the night.

    I'm using A2Hosting right now for shared, love it. Referral link.

  • NateN34NateN34 Member
    edited November 2012

    @Taz said: Liam @infinity please remove /hide ths thread. If this is a solusvm exploit, this can have hugee affect.

    Not an exploit (according to ChicagoVPS):

    "ChicagoVPS experienced a brute force on the SolusVM API for the administrative section. This caused the above affected nodes to become compromised before we were able to stop the attack."

  • jarlandjarland Administrator

    @NateN34 said: Not an exploit:

    But isn't that API locked to IP?

    I'm using A2Hosting right now for shared, love it. Referral link.

  • JackJack Member, Provider

    @jarland said: ut isn't that API locked to IP?

    It should be yes.

    Affordable Manchester (UK) Based VPS Hosting - Pure SSD OpenVZ | SSD-Cached KVM

  • ShardHostSarahShardHostSarah Member
    edited November 2012

    @jarland said: But isn't that API locked to IP?

    I know ours is, or at least SolusVM tells us it is, which is why I am asking about exploit ;)

    AnnualKVM KVM VPS from £10/year

  • TazTaz Disabled

    API can only be accessed from whmcs IP I assume . Since someone was able to.bruteforce, something might not be right?

    Time is good and also bad. Life is short and that is sad. Dont worry be happy thats my style. No matter what happens i won't lose my smile!

  • JackJack Member, Provider
    edited November 2012

    image

    Affordable Manchester (UK) Based VPS Hosting - Pure SSD OpenVZ | SSD-Cached KVM

  • jarlandjarland Administrator

    I'm sure when @CVPS_Chris gets this mess sorted he'll fill us in on whether the rest of us should be worried about it. Gonna be a long night for those guys.

    I'm using A2Hosting right now for shared, love it. Referral link.

  • CVPS_ChrisCVPS_Chris Member
    edited November 2012

    Confirmed that another host had the same issue. Everyone should be concerned.

    Dont ask who, it is up to them to release it and not my job to tell.

    ColoCrossing.com - Premier Datacenter and Cloud Solutions for Business

  • seikanseikan Member

    Luckily my VPS with them are not affected. But this is real scary! Backup, backup, backup guys!

    DNMin - Monitor All Your Domain Names at One Place

  • concerto49concerto49 Member

    Which version of SolusVM are we talking about? The latest?

    Serving you the best VPS, Web hosting, dedicated servers and more - Cloud Shards | Query Foundry
    We operate the network AS62638 | Available in Syd AU and Dallas, Los Angeles and NYC USA
  • PhilNDPhilND Member

    @CVPS_Chris

    Ego aside, I hope you are being serious about what you have just posted?

    https://nodedeploy.com | Premium VPS Solutions | Managed

  • JackJack Member, Provider

    Jesus Just spoke to Jeremiah via email this sounds nasty.

    Affordable Manchester (UK) Based VPS Hosting - Pure SSD OpenVZ | SSD-Cached KVM

  • CVPS_ChrisCVPS_Chris Member

    @PhilND said: Ego aside, I hope you are being serious about what you have just posted?

    Serious. I guess I can do one nice thing.

    ColoCrossing.com - Premier Datacenter and Cloud Solutions for Business

  • jarlandjarland Administrator
    edited November 2012

    @Jack said: Just spoke to Jeremiah via email this sounds nasty.

    What are your preliminary thoughts on the effect of revoking the API keys used for billing software? Assuming he explained more detail to you than we know. That's what I've done, as well as reduced stock to 0. I don't take chances.

    I'm using A2Hosting right now for shared, love it. Referral link.

  • JackJack Member, Provider

    @jarland said: What are your preliminary thoughts on the effect of revoking the API keys used for billing software? Assuming he explained more detail to you than we know. That's what I've done, as well as reduced stock to 0. I don't take chances.

    Disabling the API's would be best yes.

    However I wasn't 100% sure what was going on so I Just did : image

    :)

    Affordable Manchester (UK) Based VPS Hosting - Pure SSD OpenVZ | SSD-Cached KVM

  • miTgiBmiTgiB Member, Provider

    @jarland said: That's what I've done, as well as reduced stock to 0. I don't take chances.

    I think you'd be safe selling stock, just revoke terminations and do those manually if there are any after the cron run

    Hostigation High Resource Hosting - SolusVM KVM VPS / Proxmox OpenVZ VPS- Low Cost Comodo SSL Certificates
  • kbeeziekbeezie Member

    Though I'm wondering if an actual exploit occurred and if it's not say some kind of hardware failure at fault? I mean a brute force attack? Didn't have something as simple as Fail2Ban installed? Just curious cuz it doesn't seem to be adding up.

    KBeezie - Insignificant little blog about Nginx, FreeBSD, fun stuff | PhoenixVPS - Managed Support Representative
  • UGVPSUGVPS Member

    Yeah i have also followed up on this... This is BADD NEWS right now!

  • JackJack Member, Provider

    @UGVPS said: Yeah i have also followed up on this... This is BADD NEWS right now!

    ? You are the other host or what?

    Affordable Manchester (UK) Based VPS Hosting - Pure SSD OpenVZ | SSD-Cached KVM

  • jarlandjarland Administrator

    @kbeezie said: Didn't have something as simple as Fail2Ban installed?

    I don't know if fail2ban or LFD would cover SolusVM API access without some tweaks that most people would probably consider overkill prior to knowledge of such an exploit.

    I'm using A2Hosting right now for shared, love it. Referral link.

  • CVPS_ChrisCVPS_Chris Member

    @kbeezie said: hardware failure

    Hardware failure on 10 nodes at the exact same time lol? It was an exploit and when this is all over will reveal more.

    ColoCrossing.com - Premier Datacenter and Cloud Solutions for Business

  • pubcrawlerpubcrawler Banned

    Time to isolate offending IPs and start a distributed monitoring and ban of the activity. That's what providers need to band together to do in general.

    @CVPS_Chris loves --> http://www.hawkhost.com/

  • kbeeziekbeezie Member

    @jarland true without some form of configuration, depends on how the API talks to the system after login failures, but I would think there would be some kind of adjustment you could do, especially limiting access to trusted IPs (depending on who exactly they're extending the API to, or if it's just for their own apps).

    @CVPS_Chris good point, a hardware failure wouldn't knock out 10 physical nodes, but it would knock out a solus master, and if that data goes bad then it'd be a hard time re-creating the accounts (though the VPSes themselves would have still stayed up even if solus itself went down).

    KBeezie - Insignificant little blog about Nginx, FreeBSD, fun stuff | PhoenixVPS - Managed Support Representative
  • CNJeremyCNJeremy Member
    edited November 2012

    Pubcrawler any suggestions on how to prevent it or temp workarounds to prevent others from being exploited till a fix is released? Since we don't know the actual details of the exploit.

    Jeremy @ Cortex Networks LLC.

  • JackJack Member, Provider

    @CNJeremy said: Chris any suggestions on how to prevent it or temp workarounds to prevent others from being exploited till a fix is released? Since we don't know the actual details of the exploit.

    Shut down your master or disable the API

    Affordable Manchester (UK) Based VPS Hosting - Pure SSD OpenVZ | SSD-Cached KVM

  • marcmmarcm Member

    @CVPS_Chris said: Hardware failure on 10 nodes at the exact same time lol? It was an exploit and when this is all over will reveal more.

    @CVPS_Chris Considering how many of us, if not all of us are using SolusVM I guess we all wish that this wasn't an exploit. The good thing is that OpenVZ is easy to back up and I know that you guys have backup servers in the data center. Best of luck with the restoration, I hope it goes well.

  • netomxnetomx Member

    Can you please check my accoubt chris? Mine says i got blacklisted

    Referral links: DigitalOcean referral link
  • LAKidLAKid Member

    @netomx You're so funny...

  • UGVPSUGVPS Member

    Lol No im not the other host thank god.

  • JackJack Member, Provider

    @UGVPS said: Lol No im not the other host thank god.

    Ah Just the way you said it made it sound like it.

    Affordable Manchester (UK) Based VPS Hosting - Pure SSD OpenVZ | SSD-Cached KVM

  • RandyRandy Disabled
    edited November 2012

    maybe someone could get in touch with Phill @ SolusVM and ask?

  • JackJack Member, Provider

    @Randy said: maybe someone could get in touch with Phill @ SolusVM and ask?

    Already done.

    Affordable Manchester (UK) Based VPS Hosting - Pure SSD OpenVZ | SSD-Cached KVM

  • marcmmarcm Member

    @Randy I am sure that SolusVM would be extremely forthcoming with that information...

  • JackJack Member, Provider
    edited November 2012

    image

    I asked them to post here as they have an account here.

    Affordable Manchester (UK) Based VPS Hosting - Pure SSD OpenVZ | SSD-Cached KVM

  • RandyRandy Disabled

    @Jack said: Already done.

    Great, keep us updated

    @marcm said: @Randy I am sure that SolusVM would be extremely forthcoming with that information...

    lol

  • ZettaZetta Member

    @Jack Subject: URGENT EXPLOIT Priority: Medium

    Nice.

    When you find that perfect VPS, KEEP IT.

  • JackJack Member, Provider

    @Zetta said: Priority: Medium

    Well you know when you are in a rush...

    Affordable Manchester (UK) Based VPS Hosting - Pure SSD OpenVZ | SSD-Cached KVM

  • marcmmarcm Member

    @Randy said: @marcm said: @Randy I am sure that SolusVM would be extremely forthcoming with that information...

    lol

    Would you be in a hurry to tell the world that your closed source software has a major vulnerability like this?

  • RandyRandy Disabled

    i am sure it will not hurt to ask right? just to check with them, but if they do not want to release any info about the vulnerability then its their problem.

  • kbeeziekbeezie Member

    @Randy usually they don't say anything until they have a fix for it, and when they do they send out the emails to upgrade to the next version.

    KBeezie - Insignificant little blog about Nginx, FreeBSD, fun stuff | PhoenixVPS - Managed Support Representative
This discussion has been closed.