All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
BlueVM Illinois server hacked, data lost
I searched but I can't see any other thread discussing this. I received this email from BlueVM (about four hours ago, according to the timestamp).
Hello,
>
Yesterday around 4:00 PM MST one of the servers in our Illinois datacenter was compromised. This compromise lead to the loss of data on that server along with all backups and data associated with it. That data was unrecoverable and thus roughly 30 of our clients VPS are currently offline. We have no reason to believe that the data on that server was stolen by a third party. We have determined the cause of the incident to be due to an exploit carried out against our VPN services on that node. We have shut down all of our VPN services on our other nodes to prevent this incident from happening again and have changed all of our root passwords, control panel passwords and administrator logins. Clients who run a VPN on their VPS will not be affected by our shutdown of those services.
>
We are offering those clients who were/are affected by this compromise. As such we are offering our clients affected by this data loss their choice of:
>
- Three (3) months of services on a different node of their choice with the plan they currently have now.
or- A full refund for this month's services.
>
To request either of these two options simply open a new ticket with us and we will help you come up with a solution to get you back online as quickly and easily as possible.
>
In addition we are taking steps to prevent our clients from experiencing data loss again, thus starting immediately we will be offering 10 GB of free backup space with FTP, control panel and mysql access to all of our clients upon request. We feel that this change will prevent our clients from experiencing a loss of data due to hacking in the future and we are committed to helping to resolve any issues caused by this outage to the best of our ability.
>
We would like to thank each and every one of you for choosing BlueVM and we would like to assure you we are doing everything in our power to prevent an incident like this from occuring again. If you have any suggestions, comments, questions or general feedback please feel free to open a ticket with us.
>
Best Regards,
BlueVM Staff
https://bluevm.com
I'm luckily not on the affected node, and I'm sure it's little comfort to those who've lost their data, but it's really nice to see a company be upfront about what happened and the steps they've taken to ensure it doesn't happen again. Unfortunately part of the reality of the internet is that compromises do happen sometimes The compensation for the affected customers is a nice touch.
Comments
Seems like a strange incident, but it's good to see that BlueVM handled it in a reasonably professional manner. I especially like the offer of "10 GB of free backup space with FTP, control panel and mysql access to all of [their] clients upon request." Best of luck to them!
Why were the backups stored on the same server and why was there a VPN server running on the core node?
I was thinking the same.
Core node?
The core of the node. Not a container. Main root.
Were they running OpenVPN as root or an unprivileged user?
About the VPN offer, seems like someone already remind them about the hacking possibilities. But I don't remember where it ( WHT or LET.)
i have a vps with them, not sure if compromised. luckily i only use it for testing.
I have one too, the $1,5. But I don't know what to use that vps for. Since many os templates running update/upgrade is not working
The BlueVM guy doesn't seem too active on LET anymore. He used to post a lot.
@Kairus I think he has gotten enough clients to satisfy himself.
Just a guess, but at the moment I'd wager that he's too busy doing damage control to pop in for a spell.
Backups? Of $1.50/month VPSs? It's most likely just standard notification like "We have no reason to believe that the data on that server was stolen by a third party" part. If hack really happened there's of course always big reason to believe that atleast some datas are also stolen and saying anything else to calm down clients seems sort of unresponsible.
Anyway, I wish to BlueVM host all the best and hope that this will be fixed soon without too big damage for clients and company.
We had network monitoring to tell us that... No big spikes in network usage and considering there was 200 GB of data on the node between all the users there would have been some indication of a spike.
As @Aldryic said I've been preforming damage control all day... I'm at the 36 hour mark without sleep.
The VPN clients were under an unprivileged user.
Damn, best of luck.
If you get really tired I can sing to you ALL night if you want to help keep you up.
@ErawanArifNugroho what do you mean?
I can't perform #apt-get update or #apt-get upgrade, or #yum update
sudo????
LOL, nah just kidding :P
When I had the vz one I used Debian 6 without issues o_O
@ErewanArifNugroho: maybe you should have tried
emerge world
instead (gentoo joke)hehe...
How did he cause you to lose data? Did he somehow (exploit, etc) get root?
VPN clients doesn't need data o_O
Or the node had containers too?
Edit: nvm, I read again xD
This is why we send backups to Amazon S3 for safekeeping. Blow up the node, worset my customers who are not following TOS will at least have something that's 24 hours old.
Another reminder that people need to keep backups for themselves even if the provider provides it.
Self marketing at its best
Hopefully the content isnt so terrible and that amazon finds out and suspends you!
So you spend $1395.00 a month to backup just 1,000 clients or 37% of your income without paying for any S3 bandwidth? It's great your so dedicated
What part of Colorado are you from?
--
That being said I took a nap and now I feel much better.
This is what I was thinking ... Amazon does not make any sense as a back up solution for someone providing hosting solutions.
I re-did my math and assumed he only kept 3 backups on file: a daily, a weekly and a monthly... If you add in bandwidth his customers are actually saving money over choosing S3.
First thing that came to mind: