More ChicagoVPS Data Loss
Just recieved this a few mins ago. This isn't the first time this has happened, correct?
You are receiving this email because our records show that you have one or more VPS with us located on the la-vps12 node. We received an alert from our monitoring system that a drive in the RAID array of this system was in degrading condition. We verified this and contacted the Los Angeles datacenter remote hands technicians to go ahead and replace the failed HDD.
On 09-11-2013 11:27 PM EDT the remote hands technician proceeded with the hotswap replacement, however during the RAID rebuild process another hard drive failed during the live RAID rebuild process, which caused inconsistant data on the array. We have attempted to salvage the data on the RAID array and even hired external assistance to attempt to recover the server, but despite our attempts, it has been concluded that the RAID array on this server is corrupt and is in an unrecoverable state.
Unfortunately this matter was out of our hands as in the nature of technology, hardware failures can and do happen. Like everyone else, we cannot always predict when hardware may fail, however we can certainly take steps to greatly minimize the chance of something like this from happening again. We have taken the appropriate measures and have revised future node setup procedures. From now on, HDD components for all of our servers will be ordered in separate batches, so in the case of a bad shipment or HDD batch the chances of multiple drives failing is very slim. What happened with la-vps12 is very rare to see but we suspect it was due to a bad Hard Drive shipment as the HDD's on this server came from the same batch.
We have brought a brand new server online with drives from separate shipment batches and set up a replacement server for la-vps12. At the moment, we are working on reinstalling all of the VPS instances on la-vps12, your VPS should be coming back online within the next few hours as we work to reinstall all of the VPS instances hosted on this hypervisor. We appreciate your patience during this time and apologize for the inconvenience caused.