Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Scaleway's major f*ck up
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Scaleway's major f*ck up

So i was using an old VC1S instance on Scaleways system. worked fine. used it for my mailserver. Did backups manually every few and all fine.

Now all of a sudden they say the hypervisor crashed. and all my data is gone and they have no way to get any of it back besides a 1 year old snapshot....

How can a company this big have zero redundancy.. a crash of 1 system and boom all the customer data is gone.. i cant even figure out how bad of a system setup they have to run to accomplish this..

I know i probly should have used a more tightly backup regime, and looking back should have moved away from this scaleway/online.net long ago, but still.. few months of mails are gone and thats really stupid. :(

«1

Comments

  • deankdeank Member, Troll

    Do you have any plans to sue them?

  • FHRFHR Member, Host Rep

    They don't use RAID to keep prices down.

    Anyway, even if they did, anything can happen and it's absolutely your responsibility to have backups.

    If you use a mail client like Thunderbird or Outlook, just export the cached emails and upload them to your new server.

    Thanked by 1Janevski
  • TheLinuxBugTheLinuxBug Member
    edited October 2019

    Um, not really siding with them, but crashes happen, drives die... customer are expected to have their own back-ups...

    Nothing you said is either surprising or unrealistic. This happens from time to time, hardware dies unexpectedly. They probably had a big raid handling all the data and something went wrong and the raid got corrupted.

    Really the only one to blame here, sadly, is your self for not having back-ups in place. They don't promise you any data redundancy, they don't state that they provide free back-ups of the server.. so really on their side they haven't done anything wrong. In fact, at least they were honest about it and didn't leave you waiting a month before telling you you weren't getting your stuff back. They really have done everything they can really be expected to do in the situation, inform the customer and get the platform back online.

    my 2 cents.

    Cheers!

    Thanked by 1flatland_spider
  • deankdeank Member, Troll
    edited October 2019

    Well, to he Amiztly fair, OP is within his rights to be angry.

    I mean it's the host's fault after all. I just want to know whether he's suing or not because ultimately that's the only thing that matters.

    Suing them is the only way to let the host to know that he is truly angry.

    Thanked by 1Amitz
  • NeoonNeoon Community Contributor, Veteran

    Sounds more, you fucked up and blaming Scaleway for your own failure.

    Thanked by 1rm_
  • DPDP Administrator, The Domain Guy

    Ideally your data is your own responsibility and if they were really important to you, you'd manage your own backups more efficiently, so that when issues like this or any other unforeseen/unexpected issues occur, you won't be severely impacted, at least not for long.

    My two cents.

  • No backup - no sympathy.

  • rm_rm_ IPv6 Advocate, Veteran
    edited October 2019

    User's Major Fuck Up: Doesn't Backup his Mail for Months, Blames Provider

  • JordJord Moderator, Host Rep

    Ok, so they had a bit of a wobble on their servers. But in theory if you are hosting mission-critical stuff you shouldn't be using scaleway.

    If at the very least you should be taking daily backups. Backup space doesn't cost much these days. Plus there are plenty of backup solutions out there too.

    Thanked by 1sgheghele
  • @Tripleflix said:

    How can a company this big have zero redundancy.. a crash of 1 system and boom all the customer data is gone.

    Because customers didn't pay them for redundancy?

    Backups kept with the same provider aren't really backups anyway, you know?

    Thanked by 2uptime ITLabs
  • If the data was important to you, you should have backups not once but twice.

  • @Wolf said:
    No backup - no sympathy.

    If there was backups - there would be no drama.

    Here is a recipe:

    No backups
    Raid0/no raid
    No electricity redundacy
    DMCA
    pr0n

    Blemd this properly and you will get nice, healthy drama drink.

    Thanked by 2uptime level6
  • jsgjsg Member, Resident Benchmarker

    Cheap (as in low price) providers having any kind of disk redundancy (e.g. Raid), backups, dual power and or network feeds Is not something to be expected or to take for granted.

    OP wonders why a big low end provider does not have/do ... Let me turn that question around and ask OP @Tripleflix

    • Why did YOU not even make the effort to find out whether they have/do things that YOU deem important?
    • Why did YOU not make backups?
    • Why did YOU not backup or store locally all your emails?

    YOU had basically just one question in mind -> "Is it cheap?" and obviously you liked their answer - that's why you bought that VPS.

    Yes, Scaleway did f_ckup - but so did you, and the responsibility is YOURS

    Disclosure: I'm not a fan of Scaleway/Online and I too bought one of their cheap products (a small dedi) but I took care of backups because it was clear that a super cheap product from a not exactly high class provider was not to be relied on nor were I to reasonably expect good quality or good service.

    TL;DR - You want to play on the LET field? Fine, then use your brain and learn some basic rules of that field.

    Thanked by 2FHR ITLabs
  • I've been paranoid about this as well, so my solution was to set up a second VPS with a different provider and continually sync my data

  • deankdeank Member, Troll

    Cut OP some slack.

    Making backup is a very hard thing to do. The hardest part is being able to remember to make backups.
    Of course, leaving backup on the same server is another common mistake people make but you know the saying.

    The end is Amitzly nigh.

  • rm_rm_ IPv6 Advocate, Veteran

    deank said: Making backup is a very hard thing to do. The hardest part is being able to remember to make backups.

    If your backup requires you to manually run it, then you have already lost.

  • ClouviderClouvider Member, Patron Provider

    I think the issue boils down to - do you pay them for backup as a service ?

    Thanked by 1DP
  • JanevskiJanevski Member
    edited October 2019

    @FHR said:
    They don't use RAID to keep prices down.

    Anyway, even if they did, anything can happen and it's absolutely your responsibility to have backups.

    If you use a mail client like Thunderbird or Outlook, just export the cached emails and upload them to your new server.

    If I understand well, Scalway compresses/decompresses user data on permanent storage archive, then cache the data when the user server is powered on in ram on a NAS and then the user instance kernel mounts that root file system and buffers it via network from the nas.

    compression archive <-> cached nas <-> user server

    (That's how they save on drives, don't store zeroed space and only high entropy compressed data is permanently saved, which is efficient in terms of disk utilization. It's like they paid for a certain amount of disk space hardware, but they get a percentage more - compressed, than the hardware can hold - uncompressed. Sure, it costs them cpu, ram and internal bandwidth, which they can afford. Also it costs the user poweron / poweroff time and reduced reliability due to serialization, which he can afford.)

    (This is how they maintain competitive prices on storage, lesser than the hardware market dictates. I believe they even manage to make a good profit out of this practice.)

    Quite likely, the nas got rebooted and the old archived image from the last power down user server state is the only thing that remained.

    Quite potassibly, the archive has raid, but the nas ram doesn't.

    They have a whole own dev team for their system and it's something they have been working on, in house, for many years.

    Thanked by 4uptime DP ITLabs FHR
  • Two fuckups by the user's count:

    • not making backups

    • using Scaleway

    Thanked by 1uptime
  • And you expect this to have gone differently at insert-providerName in the event of a catastrophic disk/SAN failure?

    Be smart. Keep using online.net/scaleway for the cheap deals, but have an offsite backup for FUBAR-day.

    Thanked by 1uptime
  • Raid imo is pointless, just have another server for backups at a separate provider.

  • level6level6 Member
    edited October 2019

    I have an idea. Why not go on various hosting related forums to bash Scaleway, and spread the word that you don't back up your shit?

  • .

    @vimalware said:
    And you expect this to have gone differently at insert-providerName in the event of a catastrophic disk/SAN failure?

    It's been a couple years since I last sampled Scaleway's wares but back then I got the impression that they were jankier than most. So if playing fast and loose with the odds might expect chances of data loss to be a bit higher with their setup. No guarantee either way of course but from my point of view it just suggests it was indeed a double-barrelled fuckup on OP's part.

    Be smart. Keep using online.net/scaleway for the cheap deals, but have an offsite backup for FUBAR-day.

    Fair enough, can't argue with that - and yeah ... just might circle back around to see what's new with Scaleway, one of these days :smiley: They weren't that bad.

    Thanked by 2Janevski ITLabs
  • ITLabsITLabs Member
    edited October 2019

    @uptime said:
    They weren't that bad.

    Wut!? I had some issues with poor SSD performance back in 2015. About 3 years later people were still reporting the same issues. They're that bad!

    Thanked by 2uptime Janevski
  • @ITLabs said:

    @uptime said:
    They weren't that bad.

    Wut!? I had some issues with poor SSD performance back in 2015. About 3 years later people were still reporting the same issues. They're that bad!

    Other tests:
    
    dd bs=1M count=2048 if=/dev/zero of=test conv=fdatasync
    Digital Ocean (512Mb)
    2147483648 bytes (2.1 GB) copied, 10.9867 s, 195 MB/s
    
    Scaleway
    2147483648 bytes (2.1 GB) copied, 23.7442 s, 90.4 MB/s
    

    Yeah and you even used bs=1M (bull shit size 1M) and it still was not that good.
    In my experience, the mediocre I/O performance was due to the NAS overhead, plus bandwidth limitations.
    See, you are limited by your NIC, 90.4 MB/s = 723Mbps - it's the gigabit NIC.

    Their C1s were like, up 200Mbps shared for networking and up to 800Mbps shared for NAS.

    Thanked by 2uptime ITLabs
  • jsgjsg Member, Resident Benchmarker

    @SirFoxy said:
    ... just have another server for backups at a separate provider.

    2 x sh_tty VPS != safe and sound.

    My equation is more like -> If you have one or two solid systems then you can afford to have a load of dirt-cheap flaky systems too.

    Thanked by 2uptime ITLabs
  • You should always consider the worst situation... Still Scaleway is to blame...
    Maybe try Hetzner cloud with CEPH:(

  • TripleflixTripleflix Member
    edited October 2019

    yeez guys, im wasnt even bitching this hard about the backups. was more amazed at how a platform as big as scaleways can crash so hard and lose all their data..

    Everybody jumping on the "Its your own fault you didnt make backups" bandwagon.. imagine if it something like this happend to you on a day where u didnt really have the time to fix it and the provider was like; "yea sorry we lost it all, if u have a backup u can restore it now" Its still f*cking annoying

    i did have a backup but it wasnt that recent. everything is up and running on another server (not with scaleway/online.net obviously)

    @Developer_HZH

    i did just that and jumped over to hetzner could. why CEPH storage in particular thou?

  • FHRFHR Member, Host Rep

    Developer_HZH said: Maybe try Hetzner cloud with CEPH:(

    From my experience, their Ceph instances are fairly unstable; I experienced quite a few hiccups in the past months. Local storage ones are definitely better.

  • If you are going to use cheap hosting or servers, regular off-site backups are a must. Best to automate this process so you wont forget.

    People keep the prices low by using outdated and sometimes used hardware. That's how most people provide hosting services at such cheap prices. So if you want to go without backups of your data, go with hosts that do it for you. Maybe look into managed hosting.

    Thanked by 2isunbejo DP
Sign In or Register to comment.