Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


BuyVM Catastrophic Data Failure - All data lost on a node!
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

BuyVM Catastrophic Data Failure - All data lost on a node!

eric1212eric1212 Member
edited April 2018 in Providers

BuyVM suffered a catastrophic failure on a node where multiple SSD controllers failed.
Anyone else effected by this? I am! Still waiting for the new server to be provisioned. Good thing I had a semi-recent backup.
BuyVM will also be restoring their backup, but it's a few days old and unsure if it contains everything.

«134567

Comments

  • Yeah, I have backups so nothing lost apart from the downtime.

  • Inded_HostingInded_Hosting Member
    edited April 2018

    Well is the best always to keep backups from your side also to keep a backup of the backup too if is possible.

  • TomTom Member

    Is this shared?

  • deankdeank Member, Troll

    Massive hamster strike.

  • EwokEwok Member

    'Catastrophic'? That's a bit fucking dramatic.

  • @Ewok said:
    'Catastrophic'? That's a bit fucking dramatic.

    Their words, not mine.

    Thanked by 1Ewok
  • "I lost all my data and had no backup, I'm losing millions per minute"

    Posts coming in 1..2..3...

  • angstromangstrom Moderator

    Odd: so far, no announcement or tweet; network status is also good.

  • LeeLee Veteran

    Ewok said: 'Catastrophic'? That's a bit fucking dramatic.

    A bit like your reply.

  • LeviLevi Member

    Catastrophic is usual... "Colossal and unfathomable data failure" - this would be a bit better.

  • CConnerCConner Member, Host Rep
    edited April 2018

    "Help guis I am loosing a lot money my client angry at me. I make only annual backup"

  • angstromangstrom Moderator

    @Francisco, help us!

  • imokimok Member

    LTniger said: unfathomable

    Well... new word for me.

  • I do not know how it's possible. It looks like a total amateurishness....

  • angstromangstrom Moderator

    Anyone want to post the email?

  • @angstrom said:
    Anyone want to post the email?

    Join their discord, that's got a lot more info.

    Thanked by 1angstrom
  • I keep a weekly backup on the server... Is that lost as well? :D

  • How the hell do multiple controllers fail at he same time

    Thanked by 1MrH
  • WHTWHT Member

    Was is not raid10? How.the heck always fail 2 drives at once

  • angstromangstrom Moderator

    @jetchirag said:

    @angstrom said:
    Anyone want to post the email?

    Join their discord, that's got a lot more info.

    Discord, I see, but one would think that Discord shouldn't be the main channel of communication about such an event.

  • deankdeank Member, Troll

    Wanna be more dramatic?

    The end is NIIIIIGH!!!

    On a more serious note, is it for real? Could be a badly miss-timed April fool's joke.

    Thanked by 1doughmanes
  • @angstrom said:

    @jetchirag said:

    @angstrom said:
    Anyone want to post the email?

    Join their discord, that's got a lot more info.

    Discord, I see, but one would think that Discord shouldn't be the main channel of communication about such an event.

    Maybe they emailed customers. I'm not a customer of buyvm tho!

  • HBAndreiHBAndrei Member, Top Host, Host Rep

    Yeah, I'm also affected, millions just being wasted every second...

    Here's the email for those who asked for it:

  • quickquick Member

    Deadpool

  • MasonRMasonR Community Contributor

    The ponies strike again. Good luck, @Francisco! We're all pulling for a speedy and smooth recovery.

  • FranciscoFrancisco Top Host, Host Rep, Veteran

    After some looking through our invoicing the 4 drives that made this array were part of the same order batch that the lv-shared03 drives were.

    The same issue happened where the node crashed, rebooted, and the drives couldn't be seen by the BIOS.

    We tried a half dozen different nodes and 4 different HBA's in hopes of something.

    We do have recent backups so we're already restoring that. We're probably another 8 or so hours out before everything is back in action.

    We took it as a good time to swap to some nice Intel NVME's as well as upgrade the CPU's in the box.

    Not fun at all. It may very well spoil my experience with Samsung :(

    Francisco

  • FranciscoFrancisco Top Host, Host Rep, Veteran

    @Ewok said:
    'Catastrophic'? That's a bit fucking dramatic.

    It's the proper word. The array was entirely lost and we lost active data on it.

    We have backups, but in most cases it's a few days old.

    Francisco

    Thanked by 2Ole_Juul netomx
  • jetchiragjetchirag Member
    edited April 2018

    @FennecFox said:
    How the hell do multiple controllers fail at he same time

    Bad batch of SSDs (maybe)

  • deankdeank Member, Troll

    "Samsung" actually means three stars.

    What do you expect from 3-star products? :p

    Thanked by 1Francisco
  • FranciscoFrancisco Top Host, Host Rep, Veteran

    @WHT said:
    Was is not raid10? How.the heck always fail 2 drives at once

    Wasn't just 2 drives, was all 4.

    All of them are the same batch from the same seller (amazon). I've not had a chance to dig through errata's to see if there's known issues with any batches or the likes, but so far its only been 850 1TB's that have done it.

    We got other nodes on 840 1TB's and they never skip a beat.

    Francisco

Sign In or Register to comment.