Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


DigitalOcean lost our entire server
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

DigitalOcean lost our entire server

… I’m reaching out regarding your droplet. Earlier today, our Cloud Operations team was alerted to some performance issues affecting the physical server that hosts your droplet and immediately began investigating. Unfortunately, despite their recovery efforts and a filesystem check of the underlying disks, the damage was serious enough that this droplet was lost and not able to be restored.

source https://murze.be/2016/02/today-digitalocean-lost-our-entire-server/

Backup, Backup, Backup, Backup, Backup, Backup, Backup!!!

$15 credit for their mistake

Thanked by 1coreflux
«1345

Comments

  • that's "Cloud" for ya.

  • Just because it has the word "cloud" in it, doesn't make it magical. Have backups.

    Thanked by 1KuJoe
  • NexHostNexHost Member
    edited February 2016

    Run away from this thread, Runaway, Runaway, Runaway, Runaway, Runaway, Runaway!!!

  • NeoonNeoon Community Contributor, Veteran

    @tommy said:

    > … I’m reaching out regarding your droplet. Earlier today, our Cloud Operations team was alerted to some performance issues affecting the physical server that hosts your droplet and immediately began investigating. Unfortunately, despite their recovery efforts and a filesystem check of the underlying disks, the damage was serious enough that this droplet was lost and not able to be restored.
    > 

    source https://murze.be/2016/02/today-digitalocean-lost-our-entire-server/

    Backup, Backup, Backup, Backup, Backup, Backup, Backup!!!

    $15 credit for their mistake

    Just 15$? The last one got like 500$.

  • LeeLee Veteran

    Why is this even thread worthy. A server failed, WTF, really, in this day and age....

  • ClouviderClouvider Member, Patron Provider

    @TarZZ92 said:
    that's "Cloud" for ya.

    Exactly a VPS with local storage billed hourly.

  • Tagging a DO employee. @jarland

    Thanked by 1jar
  • No need to tag him, he already know :) read his comment on blog above

    Thanked by 1netomx
  • DETioDETio Member
    edited February 2016

    tommy said: … I’m reaching out regarding your droplet. Earlier today, our Cloud Operations team was alerted to some performance issues affecting the physical server that hosts your droplet and immediately began investigating. Unfortunately, despite their recovery efforts and a filesystem check of the underlying disks, the damage was serious enough that this droplet was lost and not able to be restored.

    Digital Ocean is not CLOUD, It is direct KVM through Libvirt! Unless you define cloud as on-demand infrastructure which could interpret it as. But generally it is not widely accepted for a real cloud platform to not support high availability redundancy.

    Thanked by 1Kris
  • We know :) but people really love their DO droplet.
    Cheap with many location.

  • DO IS SCAM???

  • DO is same as other 'cloud providers'

    Clean website, reliability is... eh. Not real cloud, but hey - 15 dollars!

    Spin spin spin. (PR wise)

  • It's time for...:

    $ whois digitaloceanrefugees.com
    
    Whois Server Version 2.0
    
    Domain names in the .com and .net domains can now be registered
    with many different competing registrars. Go to http://www.internic.net
    for detailed information.
    
    No match for "DIGITALOCEANREFUGEES.COM".
    
  • Aside from the fact that this was already posted in the C'est post... One VPS lost and the world goes insane!

    Thanked by 1netomx
  • rm_rm_ IPv6 Advocate, Veteran

    I'm only surprised by the measly $15, almost feels like an insult. In such cases the compensation should be $100 at least.

  • Thanked by 1netomx
  • raindog308raindog308 Administrator, Veteran

    @DETio said:
    Digital Ocean is not CLOUD, It is direct KVM through Libvirt! Unless you define cloud as on-demand infrastructure which could interpret it as. But generally it is not widely accepted for a real cloud platform to not support high availability redundancy.

    But at least it's better than OpenNebula, right?

    Thanked by 10xdragon
  • What is DO storage setup? Raid 10?

  • rds100rds100 Member
    edited February 2016

    The storage setup (raid level) is not important when your RAID card fails and screws up everything.

  • rds100 said: The storage setup (raid level) is not important when your RAID card fails and screws up everything.

    I've heard quite a few horror stories about raid card failures.
    Would software raid be more reliable?

  • Of course a less complex solution (like software RAID) could be more reliable, but software RAID can't benefit from write caching and other performance bonuses, as the hardware RAID can. Also software RAID can die a horrible death too, for instance if the motherboard fails, or the PSU fails, etc.

    Thanked by 1Geekoine
  • sb56637sb56637 Member
    edited February 2016

    In my opinion the actual news in the cited posts is actually this part:

    DigitalOcean has a paid backup service that takes weekly snapshots of all droplets. All our droplets are using that service. Now was the time to test it out. After issuing the command restoring the snapshot took about 10 minutes. When the job was finished the server was running again. It had the same IP address as before the crash and Forge (which we use to provision/manage droplets) could establish a connection again. Unfortunately the snapshot was 7 days old so all data from the past week was lost. Our client would not have been happy to hear this.

    What kind of extra-cost backup service only takes snapshots once a week, even if it is fairly economical? Back when I used to use DigitalOcean for an important website, the erratic schedule of the backup system was one of my many major complaints with them. I wasn't the only one; it was one of the earliest requests in their uservoice system, and they even declared it there to be fixed. But it looks like they've actually become more infrequent since when I used it, where at least you used to get a backup every few days. But an entire week is really negligent on their part.

    The moral of the story as the author himself mentioned is never rely on the host's backup system. If they happen to have a recent backup for you, great, it's usually the easiest to restore.

    Thanked by 1lbft
  • any recommendation for backup services for vps?

  • I think DO promotes/positions itself as a hosting solution for developers and not somewhere you should be hosting production sites.

    Isnt this true?

  • JoeMerit said: I think DO promotes/positions itself as a hosting solution for developers and not somewhere you should be hosting production sites.

    Isnt this true?

    I use Digital Ocean for production.

    The tout themselves as dev-friendly, that doesn't mean they discourage production sites. They are actually quite reliable for the price. Think of the size of DO's clientbase, then think about how often you see threads like this.

    I use the DO's weekly backups feature, but I also have daily backups to an OVH dedi. In my opinion any production server should preferably have a backup server from a different provider anyway. This way you're protected in case of either hardware/infrastructure problems, and disputes with your provider. In case a meteor nukes Softlayer SG it also gives me the added peace of mind knowing that I have backups in France.

    Thanked by 1jar
  • Geekoine said: What is DO storage setup? Raid 10?

    I think that answer lies with Jimmy Hoffa, and if Vultr does.

    Thanked by 1Geekoine
  • jarjar Patron Provider, Top Host, Veteran
    edited February 2016

    It's always good to see things through customer eyes. A RAID failure is never a good experience. A single physical server is always a potential point of failure, backups are definitely important and cannot be stressed enough.

    As for the age old "cloud" debate here, I'm going to sound like a broken record but DO is absolutely cloud. Now if you make up a new definition for "cloud" I suppose it isn't, but it is most certainly cloud.

    Really good read:
    https://vpsboard.com/topic/7485-widespread-misunderstanding-of-cloud-from-the-people-who-claim-that-there-is-a-widespread-misunderstanding-of-cloud/

    Another good read:
    http://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-145.pdf

    Cloud computing and cloud hosting are very real things. It is very important to know the difference between cloud and high availability. The reason they are closely related is that cloud is generally intended to be used to build highly available applications. Mixing up the terms basically means you won't know how to shop for what you want in hosting, and that's a big deal. The members of LET are generally great at shopping, don't let this be your weakness.

    Honestly, even I'm a little surprised that so many here are using a different definition for the word "cloud" than hosting providers themselves are using. Does that mean you guys are purchasing cloud and thinking you are purchasing high availability? That is a dangerous game :)

    @rm_ said:
    I'm only surprised by the measly $15, almost feels like an insult. In such cases the compensation should be $100 at least.

    More was given, but I'm not shouting it from the mountain tops, that was a personal decision and mostly a "thank you" for the clear view of the event through customer eyes, that will lead actual changes (the removal of the word "Booyah" to say the least). I realize that the number seems small when taken by itself, but 3x the monthly rate is an important point to remember.

    Rather than overly reward the lottery of servers being servers, I would prefer that we take this opportunity to consider how we might better educate people on horizontal scaling and backups. That's really the great thing about cloud is that you can spin up cheap servers on demand and balance your applications between them, reducing the potential impact of a single point of failure. Of course, that doesn't mean that we shouldn't be understanding and generous. Love is, after all, the primary goal. Everything else must fall under that.

    @JoeMerit said:
    I think DO promotes/positions itself as a hosting solution for developers and not somewhere you should be hosting production sites.

    Isnt this true?

    Definitely not true. We have many customers who run production services on our servers. I'm not going to name them but several of the largest ones wouldn't be too hard to find, for anyone interested. We want developers to build their applications with us, we don't want to see them leave after they finish ;)

    @Geekoine said:
    What is DO storage setup? Raid 10?

    We definitely do use RAID on all systems (not RAID0, RAID with intent for redundancy and performance). In this case it was a RAID failure.

  • Sorry to hijack this thread, but how would vultr compare to DO for production? I mean, I have some production sites on both (with backups of course) but I don't expect it to fail, like this case. Anyone has information about reliability / high availability being better or worse on vultr? How about outages on both?

  • jarland said: As for the age old "cloud" debate here, I'm going to sound like a broken record but DO is absolutely cloud. Now if you make up a new definition for "cloud" I suppose it isn't, but it is most certainly cloud.

    As ive said in the past Cloud is merely a marketing word nothing else, one could Describe my Homeserver FTP/owncloud as a cloud in some cases (as well as it's raid10)

    Take livedrive as another example.

  • jarjar Patron Provider, Top Host, Veteran
    edited February 2016

    @TarZZ92 No one would logically dispute that "cloud" cannot be applied to things that might seem silly to you, like potentially your home server configuration. It does, however, have a definition and I've given you the resources to learn that from.

    Regardless, your view of it being a marketing word is infinitely more healthy than the view that it is a more magic form of redundancy and failover than is arguably even reasonable to see in existence, because your view does not make you likely to go out and purchase something with flawed knowledge that may be dangerous for your data.

    Assuming that data is 100% safe and cannot be lost because one assumes cloud to be "without a single point of failure" is dangerous for the buyer, and for that reason alone I'll continue to point out the error. I don't want potential customers to be misled by this common opinion, ultimately leading to their choice to not take regular backups.

Sign In or Register to comment.