Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


What happens when you ask a ChicagoVPS rep about their database theft?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

What happens when you ask a ChicagoVPS rep about their database theft?

joepie91joepie91 Member, Patron Provider
edited March 2013 in General
<joepie91> oh, also, ChicagoVPS
<joepie91> now that you're here anyway
<joepie91> speaking as a CVPS customer, when can I expect to receive that advisory about customer data theft?
<joepie91> seeing as I have not seen such an e-mail yet
<hifi> so much drama in #lowendbox, would have never guessed how exciting this channel would be
<ChicagoVPS_Kevin> yup, added him to ignore

Just to give you an idea of how responsible ChicagoVPS representatives are. As a customer, this very much worries me and I would not even consider putting anything sensitive on my VPS there.

(For those somehow not aware, the ChicagoVPS SolusVM database was stolen recently, and no announcement has been sent out about this whatsoever - instead, an e-mail was sent claiming that everyones password was reset because "the passwords people use are too easily guessed".)

Comments

  • laaevlaaev Member
    edited March 2013

    Please include the full IRC conversation, instead of partial to try to make yourself look good. My IRC username somehow got changed to "ChicagoV1S_Kevin" again, and joepie91 misinterpreted my words.

    From then on, when he continued to accuse me and after his "of course" comment, I immediately added him to ignore and did not see any further messages by him afterwards. As he was on ignore, the only message I saw was hifi's, and my response directly after was in response to his comment.

    Full IRC log included below.

    11:16 < ChicagoV1S_Kevin> roberthost got hacked?

    11:16 < ChicagoV1S_Kevin> lol
    11:16 < sean> http://www.lowendtalk.com/discussion/8463/xenvz-usa-ddos-protected-vps-for-irc-with-10tb-2.50mo
    11:16 < joepie91> uses it to wipe his boxes
    11:16 < sean> better
    11:16 < ChicagoV1S_Kevin> must of missed something...
    11:16 < CheeseGamer> ChicagoV1S_Kevin: hacked, all nodes rooted, apparently wiped too
    11:16 < Red_M> UGH! java is giving me the shits
    11:16 < ChicagoV1S_Kevin> how did my name change to ChicagoV1S again...
    11:16 -!- You're now known as ChicagoVPS_Kevin
    11:16 < CheeseGamer> I was wondering about that..
    11:16 < joepie91> ChicagoV1S_Kevin: I'd recommend not being too vocal about servercrate getting owned, considering what has happened to cvps
    11:16 < joepie91> but okay
    11:17 < ChicagoVPS_Kevin> I was only asking since I wasn't on IRC earlier and didn't understand what was going on
    11:17 < ChicagoVPS_Kevin> Dont be a smart ass
    11:17 < joepie91> I was more refering to your lol
    11:17 < joepie91> that came afterwards
    11:18 < ChicagoVPS_Kevin> I was saying that since my name got changed to ChicagoV1S again
    11:18 -!- tuvxy is now known as tuv
    11:18 < ChicagoVPS_Kevin> the 'lol' was referred towards that
    11:18 < joepie91> of course.

    This is not the first time I have dealt with joepie91's immature actions, and actually had him on ignore list way before I started working for CVPS but when my IRC client changed the ignore list got wiped.

    Everyone on Lowendtalk is just bringing up past events from November for drama, and honestly its beating a dead horse. No new security incidents have happened since November 2012 and Jeremiah, Chris, and Luc already covered all bases and dealt with the ordeal in a professional manner last year. You have to understand we are not perfect but ChicagoVPS has honestly changed and grown drastically since November. With that said, anyone who has a genuine question or concern about our services can submit a support ticket. I'm not going to feed the troll and will be stepping out of this thread since @joepie91 created this thread for attention, upset that he was added to my ignore list after his "of course" comment. I usually wouldn't add someone to my ignore list over one rude remark, but I have had run-in's with him in the past along with many others, and then I realized that my IRC client never transferred the ignore list.

    P.S. If any IRC expert on here knows why my username on IRC often gets changed to "ChicagoV1S_Kevin" please let me know as I must be missing something, its getting frustrating having to change my username to ChicagoVPS_Kevin every 24 hours or so.

  • joepie91joepie91 Member, Patron Provider

    @CVPS_Kevin said: Please include the full IRC conversation, instead of partial to try to make yourself look good. My IRC username somehow got changed to "ChicagoV1S_Kevin" again, and joepie91 misinterpreted my words.

    From then on, when he continued to accuse me and after his "of course" comment, I immediately added him to ignore and did not see any further messages by him afterwards.

    Full IRC log included below.

    [snip]

    This is not the first time I have dealt with joepie91's immature actions, and actually had him on ignore list way before I started working for CVPS but when my IRC client changed the ignore list got wiped.

    If that is the case, then surely you would have no issue answering the question I asked?

  • jarjar Patron Provider, Top Host, Veteran

    When solusvm was hacked, cracked, exploited, or whatever you want to call it, wasn't the implication clear that solusvm was accessed by someone with malicious intent? That means the data was exposed. It's only logical to consider the data compromised. I don't see where another announcement is necessary. I can see how it might be preferred, but not necessary.

  • AdducAdduc Member

    @jarland said: I don't see where another announcement is necessary

    Not all of ChicagoVPS' customers frequent LET.

  • jarjar Patron Provider, Top Host, Veteran
    edited March 2013

    @Adduc If you were a customer when solusvm was accessed, you got an email. Should they have announced it to every client who signed up months afterward? Perhaps resend the email weekly? Sorry I'm just confused as to why clients who weren't around for that need to be notified about it.

  • Get your riot shield and pop corn ready.

    Grab the best sits in the house!

  • joepie91joepie91 Member, Patron Provider

    These are all of the e-mails I have received from ChicagoVPS regarding this breach:

    November 4, 2012

    Dear Customer,

    The ChicagoVPS team has been working day and night over the past 48 hours to restore our environment and mitigate the impact of the issues we’ve experienced. This evening we have doubled our support staff to help better serve the ongoing support load and to make sure we are providing the highest level of customer service possible.

    At this time all impacted nodes are back online and all customer VPS (containers) have been reinstalled using a fresh template. Our work to recover files continues; that effort is easiest for customers who made use of our centralized backup service. If you’ve used that feature please contact our support team so we may work with you to restore your files. Customers who did not utilize that method may also contact us and we will work on restoration as resources allow.

    Please understand that this is now an all-hands event and we will work diligently until every customer is online and happy. We are willing to provide whatever assistance is required, without cost, to re-setup your environments as they were before this crisis.

    Most importantly ChicagoVPS appreciates your business. We understand you have a choice when it comes to your VPS hosting and we promise to work as hard as possible to make our appreciation clear. Thank you for your patience and understanding.

    Regards,

    Chris Fabozzi

    Director Of Operations
    ChicagoVPS

    November 9, 2012

    Hello Everyone,

    I just wanted to give a quick update, since a lot of you are looking for one and had a few questions.

    First off, I want to start out by saying thank you to all of you that have been clam during this event and understand that sometimes things do happen.

    In no way, has WHMCS been effected from this, so no customer personal information such as credit cards, emails, etc. has been stolen. ChicagoVPS will also
    be implementing a regular backup service for all OpenVZ products. We will start out in Chicago and work to Buffalo, then to LA.

    We want to assure you that we are doing everything we can to make sure nothing like this can happen again, and that you can still rely on us for your hosting needs.

    If you have any additional questions, please feel free to open up a support ticket.

    Thank you all again for your business.

    Regards,

    Chris Fabozzi

    Director of Operations
    ChicagoVPS

    February 28, 2013

    Hi Sven,

    This is a service advisory notice from ChicagoVPS as we noticed you have one or more active VPS with us. As we've recently noticed an increase in customers utilizing easy to guess passwords, we are requiring all VPS control panel passwords to be reset as a precaution to protect your VPS container and its contents. We are performing a password rotation often as part of our new security policies and also to remind you as the customer to do your part in keeping your password secure and to use a complex password. We also recommend changing your passwords every few months.

    From Feb. 28, 2013 onwards, all current VPS control panel passwords have been expired. You will no longer be able to login with your old credentials, and in order to access your control panel moving forward you must access https://manage.chicagovps.net:5656/login.php and click on the "Forgot Password" link. By doing that it will send you an email with a brand new randomly generated password.

    Thank you for being a loyal customer of ChicagoVPS and for your cooperation as we do our part in keeping our users safe. If you have any questions please submit a support ticket.

    Warm Regards,

    ChicagoVPS Team

    http://www.chicagovps.net/

    Support Email: [email protected]

    Sales Email: [email protected]
    Pingdom Report: http://stats.pingdom.com/jzrszp4wfu79
    Facebook: http://www.facebook.com/chicagovps
    Twitter: http://twitter.com/chicagovps

    The conclusion:

    • There is a claim, or depending on how you interpret it, very strong implication that no customer data was stolen (see the November 9, 2012 email).
    • There is no indication that any kind of database has been accessed - there's a very big difference between "someone broke into our SolusVM and removed a bunch of VPSes" and "someone stole our SolusVM database".
    • The password reset notification is blamed on customers using "easy to guess passwords", no mention of a database breach.
    • The database has only recently started circulating in public, no notification of this was given.

    In short, this is not a sufficient notification about what exactly happened, and what information is at risk. It is unreasonable to expect people to, from these e-mails alone, conclude that their customer information is out in the wild, especially when a different reason is explicitly given for the passwords resets.

  • roberthost got hacked?
    I'd recommend not being too vocal about servercrate getting owned

    This deserves its own thread. Did servercrate send out an email notifying its customers that their data has been compromised? Did they notify their payment processors too?

  • jarjar Patron Provider, Top Host, Veteran
    edited March 2013

    Email from CVPS.

    ChicagoVPS experienced a brute force on the SolusVM API for the administrative section. This caused the above affected nodes to become compromised before we were able to stop the attack.
    
    What does this mean? Currently the VM's on these nodes are being recovered to the fullest ability of Chicago VPS staff from the incomplete data destruction process and from central backups. Any VM's unable to be recreated from the remaining data or from backups will be created fresh.
    
    ChicagoVPS is committed to customer satisfaction and any way in our ability will do what we can to get everyone back up and going as fast and as best as we can.
    
    We will post additional updates on twitter and facebook and from time to time send out an email regarding the current status of the progress.
    
    If you have any questions in the mean time, feel free to directly email me at [email protected]
    Sincerely,
    
    Jeremiah L. Shinkle
    Chief Networking Officer
    ChicagoVPS
  • unusedunused Member
    edited March 2013

    Bleh, if you don't like ChicagoVPS just cancel your service and stop using them. It's clear you aren't ever going to get the response you want.

    They are adding 20-30 new nodes per month just to keep up with new customer demand. (So they don't care about you)

  • joepie91joepie91 Member, Patron Provider

    @jarland said: ChicagoVPS experienced a brute force on the SolusVM API for the administrative section. This caused the above affected nodes to become compromised before we were able to stop the attack.

    What does this mean? Currently the VM's on these nodes are being recovered to the fullest ability of Chicago VPS staff from the incomplete data destruction process and from central backups. Any VM's unable to be recreated from the remaining data or from backups will be created fresh.

    ChicagoVPS is committed to customer satisfaction and any way in our ability will do what we can to get everyone back up and going as fast and as best as we can.

    We will post additional updates on twitter and facebook and from time to time send out an email regarding the current status of the progress.

    If you have any questions in the mean time, feel free to directly email me at [email protected]

    Sincerely,

    Jeremiah L. Shinkle

    Chief Networking Officer
    ChicagoVPS

    I have never received such an e-mail, when was it sent out?

    Aside from that, it also does not imply anything about a database being stolen, again it sounds like someone just wiped things via the SolusVM panel and that's it.

  • @joepie91 said: you that have been clam during

    Clam.

    @jarland said: twitter and facebook

    No Google Plus = Bad

  • @GIANT_CRAB said: No Google Plus = Bad

    Google plus sucks

  • jarjar Patron Provider, Top Host, Veteran

    I don't know, it was posted in that thread. We can split hairs and talk about preferences, all legitimate points, but I think the message is clear that the admin area was compromised via brute force. An unmanaged vps provider shouldn't have to dumb it down, anyone using Linux servers should know what brute force means. Compromised admin = compromised data.

  • joepie91joepie91 Member, Patron Provider

    @jarland said: I don't know, it was posted in that thread. We can split hairs and talk about preferences, all legitimate points, but I think the message is clear that the admin area was compromised via brute force.

    Do you believe it's reasonable to expect every single customer to use LowEndTalk as a notification area for things that might be threatening their privacy? I believe that's what e-mail announcements and a client area were for...

    @jarland said: An unmanaged vps provider shouldn't have to dumb it down, anyone using Linux servers should know what brute force means. Compromised admin = compromised data.

    That is a ridiculous assumption to make. In many cases bruteforcing a login to something does not result in the ability to dump the entire database or see all the information. It's also unreasonable to expect everyone with an unmanaged server to guess that their data was compromised when the issue could be resolved with one single customer-base-wide notification e-mail.

    The reality is that the most likely scenario here is ChicagoVPS trying to keep it quiet in the hopes that they won't lose customers over it. Saying "yeah well, you could have guessed it" is nothing more than an attempt to justify that dishonesty - in itself it is not a valid reason not to send an announcement out.

  • jarjar Patron Provider, Top Host, Veteran

    It was posted in that thread as a copy and paste from an email.

    Hey I'm all about jumping on screw ups. This is over, has been for months, and unmanaged Linux vps users should have enough sense to consider their information compromised if their provider has been significantly compromised. You will never type words that will please everyone.

  • mojedamojeda Member
    edited March 2013

    I can confirm I also got the email about when their solusvm was compromised.

    I'm not sure why these types of threads are even tolerated...

  • blackblack Member

    Wasn't the recent database dump from "Nov. 2012"? That happened awhile ago and people complained about it.

    People are complaining again because someone made an account on LET and started PMing DB download links?

    If they didn't publicly issue a statement then, why would they do it now? (After 3 months)

  • mojedamojeda Member
    edited March 2013

    @black said: Wasn't the recent database dump from "Nov. 2012"? That happened awhile ago and people complained about it.

    Correct, the only one that is supposedly floating around is from November 2012.

  • ZettaZetta Member

    @joepie91 said: Do you believe it's reasonable to expect every single customer to use LowEndTalk as a notification area for things that might be threatening their privacy?

    I'm not the one using IRC in place of a support ticket.

    You guys need to stop expecting high end from the low end.
    End of story.

  • blackblack Member

    @mojeda said: Correct, the only one that is supposedly floating around is from November 2012.

    In that case, all of these complaints make sense if we were all living 3 months in the past.

    I doubt they'll issue any public statements because:
    1) They haven't done it then, why would they do it now.
    2) It's be pretty silly if they did it now.

  • joepie91joepie91 Member, Patron Provider

    @mojeda said: I can confirm I also got the email about when their solusvm was compromised.

    I'm not sure why these types of threads are even tolerated...

    There is absolutely no mention of the actual customer data being compromised.

    @black said: If they didn't publicly issue a statement then, why would they do it now? (After 3 months)

    Because the data is now "in the wild" and easy to get.

    @Zetta said: You guys need to stop expecting high end from the low end.

    End of story.

    Uhm, what? Since when is paying a lower price a reason not to have normal ethical business practices?

  • jarjar Patron Provider, Top Host, Veteran

    I usually don't write about my bowel movements but if I told you what I ate today wouldn't you just assume that to be part of my day as well? Splitting hairs.

  • mikhomikho Member, Host Rep

    @joepie91
    Perhaps you should update your spamfilter not to delete emails from cvps?

    This has been delt with over and over again. This incident happened in November(?) and nothing new has come since then. Why bring it up almost 5 months later?

  • joepie91joepie91 Member, Patron Provider

    Perhaps you should update your spamfilter not to delete emails from cvps?

    As I have now said twice before, even if I had received that e-mail it would not have given me any useful information whatsoever.

    @MikHo said: This has been delt with over and over again. This incident happened in November(?) and nothing new has come since then. Why bring it up almost 5 months later?

    And this question I also already answered literally two posts above yours. Reading comprehension.

  • @MikHo said: Why bring it up almost 5 months later?

    Well, CVPS did send out an e-mail where it seems they blame their customers for having week passwords. They also waited about 4 months before resetting all passwords, at least, that's what I get from everything I read here. Those two things are troubling (if true) and deserve to be discussed.

  • jarjar Patron Provider, Top Host, Veteran
    edited March 2013

    @mpkossen To what gain though? New sys admin around has to work off what he has. Jeremiah ran the show for that time and he's been gone for a while. We know Chris isn't a sys admin. That time is passed and the person who was running the nodes is long gone.

    Just trying to be honest here from my perspective.

This discussion has been closed.