New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
ChicagoVPS hacked, bunch of VPS customers offline
This discussion has been closed.
Comments
Could be a harddrive failure.
Why are the backups on the same system and or drive as the operational system?
There are more questions than answers with this outage. We are approaching 48 hours now of downtime.
As far as the license stuff with Solus, arggh! Now I see the reason BuyVM has worked on Stallion. I say all the power to @Francisco and his company. Bring something to market Francisco. Sell the panel
Or just corner the market.
Pretty crappy what Solus did locking your license @Francisco. Doesn't give me warm fuzzies about them.
Well, I was a dick about the whole IPV6 RDNS thing. I kept bugging them in ticket to either provide me with a fix or with the source and i'd write it myself. They said no to both, so I broke out some PHP to dump their variables.
About a day later I had it fully working.
We've been asked to market stallion2 but I dunno. I'd make some quick cash upfront about it, but I think we'd end up losing something that's pretty special about us
Francisco
Why did you need their source for this, i wrote my own ipv6 rdns handling for solus without seeing a single line of their code. I still use it btw, instead of their implementation.
Would have to be on the master from the sound of it, and then I can't think of why containers on multiple nodes would be wiped. If disconnecting the master has an effect such as that, I would have to confess that I am severely unprepared and need to make some dramatic changes.
I wanted it to be seemless to the user. I could have used some redirects and such but those annoyed me.
Besides, I was up for a challange.
I did offer them a copy of the source I had written if they wanted to merge it back but they weren't interested.
Francisco
From this client's perspective, it has nothing to do with why I like your services. It's your recipe for highly sold yet highly efficient nodes that keeps people coming back I think. I think you'd only gain from it, but I'm just one opinion
Speculating here: 10 nodes went down. At least several still are down. Someone said their backups are down/broke/hosed.
Seems kind of disjointed or suspect. Don't mind my foil hat
IF master was destroyed or some central asset otherwise, I'd expect all 10 nodes to still be hosed (unless they just put them back up as new empty containers).
The master seems to be intact w/o any missing records
Doh! @Francisco
The entire situation isn't then as it has been reported. Fine to say something happened we are not sure. Even fine if CVPS says so.
I am still looking around for the wave of folks impacted. 1000 VPS'es and we have maybe 4 people between here and WHT on the threads that appear to be customers. The math doesn't add up.
Well.
They brought the master back online for everyone. Be it the 1k people that got nuked see their VM in there I don't know. They did bring the master back online pretty quick though (an hour or two?)
Fran
So they brought up 1k empty containers then?
Odd, inquired earlier today that we weren't up. Node 29 I believe.
Response from CVPS was that they were "trying" and some of the VPS'es are showing as corrupted on restore.
Make sense of all of that with your decoder ring
@Francisco @pubcrawler I was hit quite badly; lost two servers with about a month of unbackup'd data (there were central backups but CVPS has somehow lost those as well).
I was originally told that I was on an unaffected node, but it appears that's incorrect. As of right now I've heard nothing more than that they are working to get my data back, but don't know when/if it will be back.
I've been very pleased with ChicagoVPS before now; they've always been fair and capable when I've worked with them, but I am a bit concerned. I suppose only time will tell... in any case, no substitute for your own backups, I guess.
From what I heard they were spinning/have spun blank VM's up just to get people going again and are doing recovery as possible.
@Ivraatiems I wish you and everyone the best of luck getting things recovered.
Fran
The missing central backup suggest that maybe it was terminated through solus after all. Solus deletes the backups when a VPS is terminated.
Sounds like they just had some stupid API ID/Key like "winning" or had it exposed somewhere and someone used it to just cycle through container IDs and call vserver-terminate.
Welcome @lvraatiems.
My sentiments about CVPS and experience with their service has been mostly the same. Except for when we signed up and massive node abuse was ongoing and this week with Chris getting competitive, rude and then taken out by the force of digital darkness.
Can't emphasize enough to have A-B or A-B-C live systems with replication of some sort. Rsync to get started and database dumps just to get it done or just in case.
@rds100, what you propose sounds very likely.
It would be ironic if the secret key was winning
What a reversal of fate in one week for CVPS.
Hope Chris finds his humility.
Indeed, for how many VM's are wiped there are very few people talking about it, I mean look at when IPXCore was down for a bit and created the longest ever thread on LET and I don't even know if they had a thousand VM's.
@pubcrawler Right now my biggest decision to make is whether I stick with them further, or decide to try some other host. That really depends on how things go over the next few days... we'll see. I don't want to throw them out after a long period of good service but this is pretty difficult.
@nunim They're counting VMs, not customers - and if you look on their FB page, a lot more people are talking. I have two VPSes with them; others might have many more.
Update: Just got this e-mail.
>
>
>
>
>
Any provider should be giving constant updates when something like this happens, I've not seen an update on Twitter in 2 days.
Has there been more updates via email?
@Ivraatiems literally answered my question, whilst I was posting it :P
This is getting nasty. Probably they got owned and don't know what to trust and what to do first. Happened even to law enforcement, even to the point where black hats were participating in conf calls about the break-ins (if "the baddies" get a few mailboxes it's easy).
======
EDIT [ redundant email removed]
I doubt anyone actually expected to get their data back, seems to be every provider that has backups they're corrupted or unavailable when actually needed, it's just another marketing line.
@serverbear I aim to please
@nunim Usually the Central Backups work well; recently they have not been. What's bothering me more is that I was originally told I was on an unaffected node... which seems to be incorrect. Therefore, I expected to get the data on that node. I'll be reaching out to them soon to see what's going on.
Make this and other similar disasters a lesson to all: its important to make backups, its moar important testing to restore from the backups
Why is it so impossible for providers to maintain backups? Seems to just be just more marketing BS in this segment of the market.
Central backups ehh? That should be a server on private VLAN only and heavily restricted. Someone's architecture really bombed this time.
I don't expect these features though, unless a provider touts this as some value added justification.
@pubcrawler if you delete/terminate the VPS from solus, it automatically deletes the backups from the central backup server too.
Love the datacenters out there that are often cycling their generators, but never transferring load to them. Kind of like starting the car, but never moving it. Sure it runs good in place, under ideal situation (no load). But when it needs to work and move, ka-boom.
Backups are only acceptable when you, the owner of the data are doing them and responsible for them. This is why I see the segment of VPS storage as being so vital now.
@nunim, it's not a marketing line for us -- our backups are kept on completely separate systems and SolusVM has no part in the management of them. Clients can log into their backups using FTP-SSL and verify what's there.
Or maybe a staff member was on termination spree..for a couple of hours ;/
@rds100, I'd say Solus method of deleting backups and being the controller of the backups is a real umm problematic role.
Solus should be keeping all data for at least a set period of time following termination.
Not that I know, but I suspect there are data retention laws. If you have someone hosting truly illegal materials on a node, they could per-emptively kill their VPS account and thereby all the data. That certainly would irk law enforcement.
It's common with "free" hosted services to have long data retention periods even after account termination. Which laws in the US specifically mandate this are unclear to me, but there has to be something.
Does Solus have a recovery tool for deleted containers like this? If not, someone just defined the need for such.
@pubcrawler, I disagree that backups are only acceptable if you do them. Keeping your own backups is a must; however, if disaster strikes -- hopefully you can get back up and running faster with ours than your own.