Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Subscribe to our newsletter

Advertise on LowEndTalk.com

Latest LowEndBox Offers

    Scaleway's major f*ck up
    New on LowEndTalk? Please read our 'Community Rules' by clicking on it in the right menu!

    Scaleway's major f*ck up

    So i was using an old VC1S instance on Scaleways system. worked fine. used it for my mailserver. Did backups manually every few and all fine.

    Now all of a sudden they say the hypervisor crashed. and all my data is gone and they have no way to get any of it back besides a 1 year old snapshot....

    How can a company this big have zero redundancy.. a crash of 1 system and boom all the customer data is gone.. i cant even figure out how bad of a system setup they have to run to accomplish this..

    I know i probly should have used a more tightly backup regime, and looking back should have moved away from this scaleway/online.net long ago, but still.. few months of mails are gone and thats really stupid. :(

    Comments

    • deankdeank Member

      Do you have any plans to sue them?

      Select few always have issues.

    • FHRFHR Member, Provider

      They don't use RAID to keep prices down.

      Anyway, even if they did, anything can happen and it's absolutely your responsibility to have backups.

      If you use a mail client like Thunderbird or Outlook, just export the cached emails and upload them to your new server.

      Thanked by 1Janevski

      SkylonHost - affordable hourly-billed KVM VPS in Prague, CZ!
      Featuring own high performance network AS202297 | RIPE NCC member | Contact us for IPs/ASNs

    • TheLinuxBugTheLinuxBug Member
      edited October 21

      Um, not really siding with them, but crashes happen, drives die... customer are expected to have their own back-ups...

      Nothing you said is either surprising or unrealistic. This happens from time to time, hardware dies unexpectedly. They probably had a big raid handling all the data and something went wrong and the raid got corrupted.

      Really the only one to blame here, sadly, is your self for not having back-ups in place. They don't promise you any data redundancy, they don't state that they provide free back-ups of the server.. so really on their side they haven't done anything wrong. In fact, at least they were honest about it and didn't leave you waiting a month before telling you you weren't getting your stuff back. They really have done everything they can really be expected to do in the situation, inform the customer and get the platform back online.

      my 2 cents.

      Cheers!

      Thanked by 1flatland_spider

      Have an Allwinner H3 device? Android? Check out H3Droid! | Lichee Pi Zero - The 6$ SBC | #SYSarm - Get It! | Atomic Pi - $35 x86 SBC
      20+ Years IT Experience in Linux/Windows Hosting, Administration and Development

    • deankdeank Member
      edited October 21

      Well, to he Amiztly fair, OP is within his rights to be angry.

      I mean it's the host's fault after all. I just want to know whether he's suing or not because ultimately that's the only thing that matters.

      Suing them is the only way to let the host to know that he is truly angry.

      Thanked by 1Amitz

      Select few always have issues.

    • thedpthedp Member

      Ideally your data is your own responsibility and if they were really important to you, you'd manage your own backups more efficiently, so that when issues like this or any other unforeseen/unexpected issues occur, you won't be severely impacted, at least not for long.

      My two cents.

    • WolfWolf Member

      No backup - no sympathy.

    • rm_rm_ Member
      edited October 21

      User's Major Fuck Up: Doesn't Backup his Mail for Months, Blames Provider

    • JordJord Moderator, Provider

      Ok, so they had a bit of a wobble on their servers. But in theory if you are hosting mission-critical stuff you shouldn't be using scaleway.

      If at the very least you should be taking daily backups. Backup space doesn't cost much these days. Plus there are plenty of backup solutions out there too.

      BaseServ Ltd - UK Shared DirectAdmin Hosting | Litespeed + Cloudlinux + Free Backups + Free Transfers.
      BaseServ Certified to ISO/IEC 27001: 2013

    • donlidonli Member

      @Tripleflix said:

      How can a company this big have zero redundancy.. a crash of 1 system and boom all the customer data is gone.

      Because customers didn't pay them for redundancy?

      Backups kept with the same provider aren't really backups anyway, you know?

      Thanked by 2uptime ITLabs
    • If the data was important to you, you should have backups not once but twice.

      I ❤ Laravel

    • @Wolf said:
      No backup - no sympathy.

      If there was backups - there would be no drama.

      Here is a recipe:

      No backups
      Raid0/no raid
      No electricity redundacy
      DMCA
      pr0n

      Blemd this properly and you will get nice, healthy drama drink.

      Thanked by 2uptime level6

      Wordpress Hosting - Home made!

    • jsgjsg Member

      Cheap (as in low price) providers having any kind of disk redundancy (e.g. Raid), backups, dual power and or network feeds Is not something to be expected or to take for granted.

      OP wonders why a big low end provider does not have/do ... Let me turn that question around and ask OP @Tripleflix

      • Why did YOU not even make the effort to find out whether they have/do things that YOU deem important?
      • Why did YOU not make backups?
      • Why did YOU not backup or store locally all your emails?

      YOU had basically just one question in mind -> "Is it cheap?" and obviously you liked their answer - that's why you bought that VPS.

      Yes, Scaleway did f_ckup - but so did you, and the responsibility is YOURS

      Disclosure: I'm not a fan of Scaleway/Online and I too bought one of their cheap products (a small dedi) but I took care of backups because it was clear that a super cheap product from a not exactly high class provider was not to be relied on nor were I to reasonably expect good quality or good service.

      TL;DR - You want to play on the LET field? Fine, then use your brain and learn some basic rules of that field.

      Thanked by 2FHR ITLabs
    • I've been paranoid about this as well, so my solution was to set up a second VPS with a different provider and continually sync my data

    • deankdeank Member

      Cut OP some slack.

      Making backup is a very hard thing to do. The hardest part is being able to remember to make backups.
      Of course, leaving backup on the same server is another common mistake people make but you know the saying.

      The end is Amitzly nigh.

      Select few always have issues.

    • rm_rm_ Member

      deank said: Making backup is a very hard thing to do. The hardest part is being able to remember to make backups.

      If your backup requires you to manually run it, then you have already lost.

    • ClouviderClouvider Member, Provider

      I think the issue boils down to - do you pay them for backup as a service ?

      Thanked by 1thedp

      Clouvider Leading UK Cloud Hosting solution provider || UK Dedicated Servers Sale || Tasty KVM Slices || Latest LET Offer

      Web hosting in Cloud | SSD & SAS True Cloud VPS on OnApp | Private Cloud | Dedicated Servers | Colocation | Managed Services

    • JanevskiJanevski Member
      edited October 21

      @FHR said:
      They don't use RAID to keep prices down.

      Anyway, even if they did, anything can happen and it's absolutely your responsibility to have backups.

      If you use a mail client like Thunderbird or Outlook, just export the cached emails and upload them to your new server.

      If I understand well, Scalway compresses/decompresses user data on permanent storage archive, then cache the data when the user server is powered on in ram on a NAS and then the user instance kernel mounts that root file system and buffers it via network from the nas.

      compression archive <-> cached nas <-> user server

      (That's how they save on drives, don't store zeroed space and only high entropy compressed data is permanently saved, which is efficient in terms of disk utilization. It's like they paid for a certain amount of disk space hardware, but they get a percentage more - compressed, than the hardware can hold - uncompressed. Sure, it costs them cpu, ram and internal bandwidth, which they can afford. Also it costs the user poweron / poweroff time and reduced reliability due to serialization, which he can afford.)

      (This is how they maintain competitive prices on storage, lesser than the hardware market dictates. I believe they even manage to make a good profit out of this practice.)

      Quite likely, the nas got rebooted and the old archived image from the last power down user server state is the only thing that remained.

      Quite potassibly, the archive has raid, but the nas ram doesn't.

      They have a whole own dev team for their system and it's something they have been working on, in house, for many years.

      Thanked by 4uptime thedp ITLabs FHR

      You are dreaming. | And it's a nightmare. | THE SECRET THREAD | THE TRUTH | HAVES YOU SEEN THIS YURA?

    • Shot2Shot2 Member

      Two fuckups by the user's count:

      • not making backups

      • using Scaleway

      Thanked by 1uptime

      Providing less than /64 means "we are clueless about IPv6". I haz Aruba, IonSwitch, OneProv, Veesp.

    • And you expect this to have gone differently at insert-providerName in the event of a catastrophic disk/SAN failure?

      Be smart. Keep using online.net/scaleway for the cheap deals, but have an offsite backup for FUBAR-day.

      Thanked by 1uptime

      Down with the sic_K_ness...
      The doctor might be going K-razy with the low end spirit. 😉

    • Raid imo is pointless, just have another server for backups at a separate provider.

      lurking in the shadows like a wombat or some shit

    • level6level6 Member
      edited October 21

      I have an idea. Why not go on various hosting related forums to bash Scaleway, and spread the word that you don't back up your shit?

      That's not me.

    • .

      @vimalware said:
      And you expect this to have gone differently at insert-providerName in the event of a catastrophic disk/SAN failure?

      It's been a couple years since I last sampled Scaleway's wares but back then I got the impression that they were jankier than most. So if playing fast and loose with the odds might expect chances of data loss to be a bit higher with their setup. No guarantee either way of course but from my point of view it just suggests it was indeed a double-barrelled fuckup on OP's part.

      Be smart. Keep using online.net/scaleway for the cheap deals, but have an offsite backup for FUBAR-day.

      Fair enough, can't argue with that - and yeah ... just might circle back around to see what's new with Scaleway, one of these days :smiley: They weren't that bad.

      Thanked by 2Janevski ITLabs

      the Amitz.party lives on!

    • ITLabsITLabs Member
      edited October 22

      @uptime said:
      They weren't that bad.

      Wut!? I had some issues with poor SSD performance back in 2015. About 3 years later people were still reporting the same issues. They're that bad!

      Thanked by 2uptime Janevski
    • @ITLabs said:

      @uptime said:
      They weren't that bad.

      Wut!? I had some issues with poor SSD performance back in 2015. About 3 years later people were still reporting the same issues. They're that bad!

      Other tests:
      
      dd bs=1M count=2048 if=/dev/zero of=test conv=fdatasync
      Digital Ocean (512Mb)
      2147483648 bytes (2.1 GB) copied, 10.9867 s, 195 MB/s
      
      Scaleway
      2147483648 bytes (2.1 GB) copied, 23.7442 s, 90.4 MB/s
      

      Yeah and you even used bs=1M (bull shit size 1M) and it still was not that good.
      In my experience, the mediocre I/O performance was due to the NAS overhead, plus bandwidth limitations.
      See, you are limited by your NIC, 90.4 MB/s = 723Mbps - it's the gigabit NIC.

      Their C1s were like, up 200Mbps shared for networking and up to 800Mbps shared for NAS.

      Thanked by 2uptime ITLabs

      You are dreaming. | And it's a nightmare. | THE SECRET THREAD | THE TRUTH | HAVES YOU SEEN THIS YURA?

    • jsgjsg Member

      @SirFoxy said:
      ... just have another server for backups at a separate provider.

      2 x sh_tty VPS != safe and sound.

      My equation is more like -> If you have one or two solid systems then you can afford to have a load of dirt-cheap flaky systems too.

      Thanked by 2uptime ITLabs
    • You should always consider the worst situation... Still Scaleway is to blame...
      Maybe try Hetzner cloud with CEPH:(

    • TripleflixTripleflix Member
      edited October 22

      yeez guys, im wasnt even bitching this hard about the backups. was more amazed at how a platform as big as scaleways can crash so hard and lose all their data..

      Everybody jumping on the "Its your own fault you didnt make backups" bandwagon.. imagine if it something like this happend to you on a day where u didnt really have the time to fix it and the provider was like; "yea sorry we lost it all, if u have a backup u can restore it now" Its still f*cking annoying

      i did have a backup but it wasnt that recent. everything is up and running on another server (not with scaleway/online.net obviously)

      @Developer_HZH

      i did just that and jumped over to hetzner could. why CEPH storage in particular thou?

    • FHRFHR Member, Provider

      Developer_HZH said: Maybe try Hetzner cloud with CEPH:(

      From my experience, their Ceph instances are fairly unstable; I experienced quite a few hiccups in the past months. Local storage ones are definitely better.

      SkylonHost - affordable hourly-billed KVM VPS in Prague, CZ!
      Featuring own high performance network AS202297 | RIPE NCC member | Contact us for IPs/ASNs

    • somiksomik Member

      If you are going to use cheap hosting or servers, regular off-site backups are a must. Best to automate this process so you wont forget.

      People keep the prices low by using outdated and sometimes used hardware. That's how most people provide hosting services at such cheap prices. So if you want to go without backups of your data, go with hosts that do it for you. Maybe look into managed hosting.

      Thanked by 2isunbejo thedp

      Don't be so serious. It's just a forum. No one cares what you think anyway.

    • jsgjsg Member

      @somik

      No. Old hardware usually is not the problem. The problem is that very cheap hosting tends to be based on single power supply, non Raid (and/or with worn out battery), (often heavily) used hard drives, etc.
      That is what makes things flaky, not an E26xx v3 vs current or DDR3 vs DDR4.

    • somiksomik Member

      @jsg said:
      @somik

      No. Old hardware usually is not the problem. The problem is that very cheap hosting tends to be based on single power supply, non Raid (and/or with worn out battery), (often heavily) used hard drives, etc.
      That is what makes things flaky, not an E26xx v3 vs current or DDR3 vs DDR4.

      Agree with you there. Thats what I meant with "outdated and used". I did not consider redundent power supply or RAID drives though. Good point there.

      Don't be so serious. It's just a forum. No one cares what you think anyway.

    • Distributed systems are Hard.

      If you don't understand all the SPOFs in your provider's kit, make sure you're ready to recover from every conceivable one, on your own.

      @Developer_HZH said:
      Maybe try Hetzner cloud with CEPH:(

      I wouldn't run anything off Ceph exclusively either(without an offsite backup on non-ceph storage.)

      All it takes is a software update or a public CVE to screw your happiness.

      Down with the sic_K_ness...
      The doctor might be going K-razy with the low end spirit. 😉

    • Hello @Tripleflix,

      I’m Thomas, Product Marketing Manager at Scaleway. I’m truly sorry about this incident. We work hard to provide a stable and reliable service, but despite our vigilance, crashes do happen as some have said before.

      VC1 is an old generation of instances and doesn’t offer RAID indeed, but our current Development and General Purpose Instances do include replicated storage by default. We also have an automatic backup feature in the pipeline that should make disaster recovery easier.

      We will be happy to let you test our DEV1-S for free as a compensation. Please do not hesitate to contact our assistance team, we’ll answer you shortly.

    • somiksomik Member

      @thomasschmit said:
      Hello @Tripleflix,

      I’m Thomas, Product Marketing Manager at Scaleway. I’m truly sorry about this incident. We work hard to provide a stable and reliable service, but despite our vigilance, crashes do happen as some have said before.

      VC1 is an old generation of instances and doesn’t offer RAID indeed, but our current Development and General Purpose Instances do include replicated storage by default. We also have an automatic backup feature in the pipeline that should make disaster recovery easier.

      We will be happy to let you test our DEV1-S for free as a compensation. Please do not hesitate to contact our assistance team, we’ll answer you shortly.

      Ah, a proper reply from a host.

      Starts reply with an apology.

      Explains the situation and what actions are being taken to prevent this in future.

      Offers a proper compensation that is not a "discount coupon".

      Many hosts can learn from this post.

      Don't be so serious. It's just a forum. No one cares what you think anyway.

    • Shot2Shot2 Member

      Still, that'd be Scaleway. Horrible templates, contorted IPv4, fubar IPv6.

      Providing less than /64 means "we are clueless about IPv6". I haz Aruba, IonSwitch, OneProv, Veesp.

    • Thanked by 2ITLabs alilet

      the Amitz.party lives on!

    • How much were you paying for your Scaleway VPS?

    • @FHR said:

      Developer_HZH said: Maybe try Hetzner cloud with CEPH:(

      From my experience, their Ceph instances are fairly unstable; I experienced quite a few hiccups in the past months. Local storage ones are definitely better.

      I just used their ceph plan for a short while and didn't notice any hiccups... So thanks for your experience, I would not recommend others to try their ceph plan anymore if high availability required.

    • dragon1993dragon1993 Member
      edited October 23

      @Developer_HZH said:

      @FHR said:

      Developer_HZH said: Maybe try Hetzner cloud with CEPH:(

      From my experience, their Ceph instances are fairly unstable; I experienced quite a few hiccups in the past months. Local storage ones are definitely better.

      I just used their ceph plan for a short while and didn't notice any hiccups... So thanks for your experience, I would not recommend others to try their ceph plan anymore if high availability required.

      I migrate last month NVMe to Ceph, i got kernel panic one time. (storage issue), but performace it seems better. (ioping/dd bench) @Hetzner_OL

    Sign In or Register to comment.