Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


DigitalOcean is a joke!
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

DigitalOcean is a joke!

hdpixelhdpixel Member
edited May 2013 in General

After 12 days of signing up with them, they lost all my data. I have a back up so I am not worry about that. However, I don't trust them anymore.

Be careful if you sign up with them. They credited $50 on my account.

Droplet History
Event Initiated Execution Time
Vnc Enable 8 minutes ago 3.0 Seconds
Create 12 days ago 111.0 Seconds

Link to the support page.
!(https://www.diigo.com/item/image/3x1v0/cvaj)

This is what they wrote.

Unfortunately it looks like the hypervisor that the server is on has failed. We are investigating this issue with the manufacturer.

We are unable to recover the system so if you have any backups or snapshots of that server please create a new virtual server from them.

I am sincerely sorry about this!

We are not taking this incident lightly! We have also issued you an SLA credit due to this event.

Thanks,

Etel

«13

Comments

  • CoreyCorey Member

    You kept backups, and it looks like they handled the situation in the best way possible. What else could they have done?

  • alexalex Member

    @hdpixel said: However, I don't trust them anymore.

    Why would you "trust" a company, especially in LEB budget?

    Thanked by 1coreflux
  • marcmmarcm Member

    @hdpixel - We had 2 power DNS nodes with them, I killed them both yesterday. Had the $50 incident as well when they killed my VPS... I was lucky because the main domain database is in our data center in Phoenix, so no real data was lost... but common, seriously? For really good SSD hosting I recommend RamNode.

  • bcrlsnbcrlsn Member

    I kinda feel like DO is experimenting all the time as well. They have good intentions but not very good execution of service.

    Thanked by 1coreflux
  • HalfEatenPieHalfEatenPie Veteran
    edited May 2013

    I feel like they did what they could.

    They had an issue. This created dataloss. They tried to recover the data but failed. They have stated they're not taking the incident lightly and promise to look into it (and assumption is that they'll try to minimize this from happening again). They have given you 50 dollars credit.

    You have the misfortune of signing up with them 12 days before this incident happened. I'm sure they had other customers who were affected too. You maintained proper backups (therefore exercised smart data safety/retention procedures).

    Granted what they (Digital Ocean) could have done was maintain backup nodes for each servers but that depends on their business plan and if they can maintain it.

    I personally see this as an issue that has happened (physically) and that Digital Ocean did the proper measures to fix it (in the short term, lets see how they improve it in the long term).

    Thanked by 1coreflux
  • IntcsIntcs Member
    edited May 2013

    @Corey said: You kept backups, and it looks like they handled the situation in the best way possible. What else could they have done?

    Does DigitalOcean claim to have auto backups? As I'm finding a hard time seeing a provider that's honest in taking backups :|

  • marcmmarcm Member

    @bcarlsonmedia - I feel the same way about them. I don't mind the competition, especially since at least I am a small business trying to carve a niche market for myself. However I think that DO made a few wrong choices, especially the choice to implement KVM in a cloud environment by themselves. DOs' response in OP seems like a blanket BS statement.

    Unfortunately it looks like the hypervisor that the server is on has failed. We are investigating this issue with the manufacturer.

    I wonder how exactly the KVM hypervizor fails. I've been working with KVM for about a year, tested it, tried to crash nodes by overloading them, etc. and it never gave up once. This was during initial testing before we started offering KVM VPS servers. Contacting the manufacturer? Who, Red Hat?

    We are unable to recover the system so if you have any backups or snapshots of that server please create a new virtual server from them.

    Supposedly the Hypervisor failed, it still wouldn't wipe out the LVM with the instance on it. Sounds like BS. How about "we accidentally deleted your data"?

    I can say two things to Digital Ocean:
    1) Xen PV would have probably been a better choice for a custom cloud environment. Easier to create backups and roll out instances.

    2) Don't give customers BS and blanket responses because many of them are actually educated when it comes to matters of hosting.

  • ATHKATHK Member

    @HalfEatenPie said: Granted what they (Digital Ocean) could have done was maintain backup nodes for each servers but that depends on their business plan.

    They are unmanaged, I feel that it's up to the customer, especially since they give you the tools to make a automatic backups and restore them...

  • danodano Member

    I kinda feel like DO is experimenting all the time as well.

    Agree with this statement entirely -- I feel many brands here on LET that have more experience and have ran more reliable operations than DO. I am pretty certain that DO has VC money, but they need to quite spending it all on marketing and figure out how-to not lose data so frequently.

    Thanked by 1coreflux
  • unusedunused Member

    @marcm said: I wonder how exactly the KVM hypervizor fails. I've been working with KVM for about a year, tested it, tried to crash nodes by overloading them, etc. and it never gave up once. This was during initial testing before we started offering KVM VPS servers. Contacting the manufacturer? Who, Red Hat?

    Maybe their SSD's went read-only/raid10 broke etc. Bound to happen (at least the read only issue)

    Surprised that there are multiple reports of impact if this was just a single node issue. DO must have hundreds of physical boxes.

    I really enjoy using DO for testing, but they just don't feel right for something I want to trust 24x7.

    They do have a backup feature as well, but I suspect they are local snapshots. In a case like this, snapshots are worthless. The old lesson of taking your own offsite backups still hold true!

  • upfreakupfreak Member

    @Intcs said: Does DigitalOcean claim to have auto backups?

    They seem to offer weekly auto-backup which is optional and costs 20% of the monthly server cost ( a $5 server if backed up costs $1 extra per month)

  • marcmmarcm Member
    edited May 2013

    I assume that the snapshots are physical copies of the LVM volumes (kind of like dd), so that has to be slow, even on SSD. That's why I mentioned Xen PV, because with Xen PV you can just do a simple tar.gz of the instance and you're done.

  • ATHKATHK Member

    Only thing about RamNode is this

    Client agrees not to run any programs on his or her VPS which use 90% of any CPU core for an extended period of time or which cause a high CPU load on the host node for an extended period of time. In other words, client agrees not to use the equivalent of a full core for him or herself for an extended period of time.

    Where as DO doesn't list anything as such.

  • @ATHK said: They are unmanaged, I feel that it's up to the customer, especially since they give you the tools to make a automatic backups and restore them...

    Yeah but still you can be an unmanaged service and still take backups if you make sure your operation can support it (e.g. in your business plan).

    cough Catalyst does weekly backups cough

  • marcmmarcm Member

    @ATHK The only reason why RamNode has this listed is because they assign a high core count for their KVM and OpenVZ VPS servers, so of course you wouldn't want someone to do Bitcoin mining full time on a Xeon E3 node. The other option is to limit the number of cores and remove that clause. Either option is valid, however RamNode choose to give customers more bang for their buck.

  • Nick_ANick_A Member, Top Host, Host Rep
    edited May 2013

    @ATHK said: Only thing about RamNode is this

    It's to ensure CPU power is there when clients need it, but not to be hogged all day every day. If you need a full dedicated core on an E3-1240v2, you need a dedicated server. You should not expect constant usage of full cores to yourself on a high performance system in our price range.

  • marcmmarcm Member
    edited May 2013

    @Nick_A said: It's to ensure CPU power is there when clients need it, but not to be hogged all day every day. If you need a full dedicated core on an E3-1240v2, you need a dedicated server.

    Still, it's pretty generous to allocate all cores to most VPS packages.

  • ATHKATHK Member

    I don't expect anything, was just stating that DO don't list it in the TOS or anywhere else that I can see, which I assume would let me hog my core.

  • jarjar Patron Provider, Top Host, Veteran
    edited May 2013

    @hdpixel said: They credited $50 on my account.

    I would agree that this is sufficient and generous. Data loss sucks. Digital Ocean isn't one of the oldest VPS providers or anything, they're just one of the better funded startups and they have a very attractive model for the least effort to getting a KVM VPS online in a user friendly way. I'm sure that given time they will learn the lessons that older providers have. If they improve as they go, they have potential to be the unmanaged version of rackspace in my opinion.

    Thanked by 1coreflux
  • marcmmarcm Member

    @jarland said: I'm sure that given time they will learn the lessons that older providers have.

    @jarland - Couldn't have said it better myself. We learned some KVM lessons the hard way as well, especially when it comes to tuning I/O.

  • unusedunused Member
    edited May 2013

    @marcm - I'm sure DO uses 8x or 10x ssd in their nodes w/ 256 or 512gb of ram.

    IMHO, the only thing that nags about @Nick_A / ramnode is that he is using such cheap CPU's - max 32gb of ram - and it's basically mid-level hardware. Makes me wonder about the SSD's. As he's mentioned he goes for the faster clock speed which I'm sure doesn't hurt benchmarks and he's doing a great job optimizing his service.

    Not knocking ramnode, they have a great service and I like my yearly openvz plan.. their KVM plans may be different. There's just a feeling of having $10k+ boxes vs $3-4k boxes.

    On the other hand (i've mentioned before) while the DO interface is unique, experience/track record hasn't been steller. It seems like a great service for temporary vms but I haven't felt like I would trust them with something that is going to power anything that requires 24x7 uptime --- from backups/snapshots disappearing, to failed nodes, to price re-adjustments to various weirdness in their ui/stuck jobs etc. now and then they just don't feel 100% "solid."

    I don't like solusvm for a panel either. I think each have their strengths on the LEB side. I'd say use DO for anything that you need hourly billing for and Ramnode is a great choice for something more stable/performance oriented (that isn't too cpu heavy)

  • marcmmarcm Member

    @unused said: IMHO, the only thing that nags about @Nick_A / ramnode is that he is using such cheap CPU's - max 32gb of ram - and it's basically consumerish hardware. Makes me wonder about the SSD's. As he's mentioned he goes for the faster clock speed which I'm sure doesn't hurt benchmarks and he's doing a great job optimizing his service.

    @unused - While I won't list the hardware that @Nick_A is using, suffice to say that there is nothing "consumerish" about it. E3 CPUs perform far better for many tasks than E5 CPUs. I can tell you one thing about @Nick_A: he doesn't even wanna consider using anything but server grade hardware in his nodes. You can rest assured that the SSDs are high quality as well.

    @unused said: Not knocking ramnode, they have a great service and I like my yearly openvz plan.. their KVM plans may be different. There's just a feeling of having $10k+ boxes vs $3-4k boxes.

    You just knocked them and who has $10K+ nodes anyway?

  • miTgiBmiTgiB Member

    @unused said: There's just a feeling of having $10k+ boxes vs $3-4k boxes.

    You can give some examples? The E3's @Nick_A uses are very common hypervisor boxes for the cloud providers, and the new nodes he is deploying are E5's very similar to what I've been using, and they are not $10k boxes, but closer to $6k, so if he is using the same/similar hardware as the front end hypervisors to the cloud providers, what is the concern? Cloud providers are going to have storage boxes which is different, but seems to be the downfall of most cloud providers, as we seem to see them all lose customer data every 6 months or so.

  • unusedunused Member
    edited May 2013

    @marcm said: You just knocked them and who has $10K+ nodes anyway?

    Not at all, I've had nothing but positive comments for ramnode and have two active plans with them. Just because they've optimized to use mid-level equipment (i guess i7's would be considered consumer) and use openvz to oversell like everyone else doesn't = a negative.

    Frankly with @Nick_A does what he does here better then almost everyone else - he isn't just reselling on dedicated servers and provides a rock solid, optimized service with great performance.

  • rm_rm_ IPv6 Advocate, Veteran
    edited May 2013

    @hdpixel said: They credited $50 on my account.

    No other company in LEB or no LEB would probably to even come close to doing something similar to this "SLA" response. Even Linode would probably just tell you "sorry and f. u." in case of any problem. From some of the "established LEB providers" I did not get even a simple "sorry" in case of stupid downtimes because of a simple human error on the provider part. Soooooooooo I don't think it's fair that after this you also slander them publicly. As with any hoster, you should have kept your own daily backups. And unless you were running some serious commercial stuff (e.g. an E-commerce website/shop - and were you?) a loss of day's worth of data most likely did not cost you anywhere close to $50.

  • Nick_ANick_A Member, Top Host, Host Rep

    @unused said: There's just a feeling of having $10k+ boxes vs $3-4k boxes.

    I appreciate your positive comments--I just want to clarify one thing. I think your perspective is a bit off here. Spending $3k per E3 node is probably a rarity in this industry. That is very expensive if you consider you are only getting 32GB RAM of clients on it. It's a good thing rather than a bad thing for clients.

    I'd be happy to use larger, more expensive setups for SSD KVM, but the performance is much better on smaller ones. Trust me - I wasted a few thousand on various 2U chassis to try to make an SKVM-E5 line to match our very popular CKVM-E5 (by the way, ~$8k per node there). But the I/O was slower than loading up our E3-1240v2s for SKVM. Maybe I didn't find the exact right way to do it; who knows? But the reason I use E3s for OpenVZ and KVM SSD is not to be cheap. I do it because it maximizes performance for our clients.

    Either way, it's all about the ratio of RAM/SSD/CPU, rather than how much each node costs. Loading up E5s for KVM makes more money than using E3s, but we only use E5s when performance is the same or better than our E3s. We went with E5s-only (E5-2630s) for SSD-Cached because i. So you spend more but really end up making a bit more relatively speaking. Spending more doesn't make it a better server for clients or more "professional grade". It just means you bought a bigger server with more in it.

    What SSDs does DO use? What about other SSD VPS hosts? I doubt any of them in the low end market use anything but consumer SSDs (Intel 335/520s, Samsung 830/840Ps, are technically all consumer grade).

    Thanked by 1coreflux
  • DStroutDStrout Member

    @unused said: I really enjoy using DO for testing, but they just don't feel right for something I want to trust 24x7.

    My feeling exactly. While I do think DO responded appropriately, I would have just rolled my eyes and moved on. I've had few issues with them (just one, really), but I've heard of some other problems that didn't affect me. By taking the same position as @unused, I'm in good shape if I do have something fail, because it almost certainly wasn't critical.

  • jarjar Patron Provider, Top Host, Veteran

    Off topic, but @Nick_A chooses a lower capacity on nodes than a lot of us and it pays off. He has some of the most consistent performance, if not the most, in the price bracket. Heck, probably in several price brackets above his highest non promo package. I'm not sure I'd over think it much, I just praise his ability to run a node. Takes a lot to say, as a provider proud of his own services, that I'd pick RamNode before myself.

  • unusedunused Member
    edited May 2013

    @Nick_A that's very true - perhaps I'm assuming the typical 5-6x ovz overselling and the thought of that on 32gb gave me pause.

    Then again, having many smaller nodes with less customers isn't a bad approach either, and as I've mentioned ramnode performance & track record really can't be beat. If I was getting into the LEB business I'd use you as the yardstick.

    @jarland @DStrout - in the whole DO/other conversation here I think you've really nailed it: consistency -- I've used them extensively and they just don't have it. Great for testing and temporary vms, but that's all in my experience.

  • Nick_ANick_A Member, Top Host, Host Rep

    @unused said: @Nick_A that's very true - perhaps I'm assuming the typical 5-6x ovz overselling and the thought of that on 32gb gave me pause.

    Right, gotcha. If we were doing that kind of overselling, I would be very worried as well! But our commitment is to performance before anything else.

Sign In or Register to comment.