Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Letbox terminated client's server by mistake and lose all data
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Letbox terminated client's server by mistake and lose all data

lightforcelightforce Member
edited January 2021 in General

Jesus, great work for their 9 anniversary. I'm unfamiliar with whmcs, could that be happened?
They claimed they outsourced some exams from some team. And credit for one more month for this case.
SO BE CAREFUL with YOUR OFFICIAL PROJECT, this is not a honoured provider

Thanked by 1JasonM
«134

Comments

  • So, they are offering no solution to restore at all? they deleted - didnt suspend? thats one fucked up procedure, theres no excusing that at all.

    In my mind, financial compensation is the only remedy here, but thats unlikely.

    I would move away from them, yesterday.

    oh, and backups ofc.

    Thanked by 1Chronic
  • @lightforce I think examination means while checking / inspection / testing / validating / verfying or something like that.

    Atleast in the context of the thread it sounds like that, it cannot be a test ( Like a school test ).

  • DPDP Administrator, The Domain Guy

    It's simple - sh*t happens.

  • @lightforce said: SO BE CAREFUL with YOUR OFFICIAL PROJECT

    OFFICIAL projects should have MULTIPLE OFFICIAL backups.

  • This is why you should always back up your data, if it's important enough. Never know when mistakes like this will happen.

    The provider should of had some sort of retention policy in place, like when the VM service is terminated wait 3 days before deleting the data permanently. I've heard of providers doing this just in case something like this happens.

  • Probably million tabs was opened in browser and he pushed delete in the wrong one. It happens. But there is no backups at provider side?!

  • I did back my database, due to the server had lots of 4k tiny files, backup will utilize much resources, so I backuped manually on schedule. Now the site could not be retained because DB and site file snap is not the same time.

    @Trav said:
    This is why you should always back up your data, if it's important enough. Never know when mistakes like this will happen.

    The provider should of had some sort of retention policy in place, like when the VM service is terminated wait 3 days before deleting the data permanently. I've heard of providers doing this just in case something like this happens.

  • Just one more month for compensation. I have not yet made decision as refund or continue to use.

    @Adam1 said:
    So, they are offering no solution to restore at all? they deleted - didnt suspend? thats one fucked up procedure, theres no excusing that at all.

    In my mind, financial compensation is the only remedy here, but thats unlikely.

    I would move away from them, yesterday.

    oh, and backups ofc.

  • yoursunnyyoursunny Member, IPv6 Advocate

    @lightforce said:
    Now the site could not be retained because DB and site file snap is not the same time.

    The lesson is: have only one source of truth, and keep it in an atomic unit.

    For my website, the source of truth is the git repository. There's no database, and the PHP scripts are not allowed to create files other than in an ephemeral cache folder that is lost upon reboot. All the content is committed to the git repositories and then rsync'ed to the server.
    Having adopted this system, a recovery takes less than an hour without provider's help.

    https://yoursunny.com/t/2017/yoursunny-com-git/

    Thanked by 2nyamenk ariq01
  • @lightforce said:
    Just one more month for compensation. I have not yet made decision as refund or continue to use.

    @Adam1 said:
    So, they are offering no solution to restore at all? they deleted - didnt suspend? thats one fucked up procedure, theres no excusing that at all.

    In my mind, financial compensation is the only remedy here, but thats unlikely.

    I would move away from them, yesterday.

    oh, and backups ofc.

    I don't find that would be a good compensation for a month caused all data lose. i will never accept this offer.

  • @snt said:
    this guy will promote his website in every possible thread he comment in

    He's doing push-ups for that. And a push-up this year is a new black.

    Thanked by 1yoursunny
  • @LTniger said:

    @snt said:
    this guy will promote his website in every possible thread he comment in

    He's doing push-ups for that. And a push-up this year is a new black.

    he is just full of himself thats all.. and im sure other people here agree to this

  • @snt said:

    @LTniger said:

    @snt said:
    this guy will promote his website in every possible thread he comment in

    He's doing push-ups for that. And a push-up this year is a new black.

    he is just full of himself thats all.. and im sure other people here agree to this

    sunny is cool dude. Chill, community is alright.

  • cazrzcazrz Member
    edited January 2021

    Not good.

    Modulesgarden should have worked and test things on staging environment and not on production server. Why the hell will you test the terminate feature on a production server/site.

    Anyway shit happens, just like when the engineer of AWS is playing with the playbook which is actually connected on the production block storage. They also deleted some data and no way to put back. IIRC

    Now, My question is. When using block storage in Letbox and this kind of shit happens again, does it mean that the data in block storage also gets automatically deleted when the account of server gets terminated?

    IIRC, the data is preserved when you reinstall the server using ISO and not template. When you reinstall server using template then the block storage data gets deleted. I haven't tried the terminate server process yet.

    I think I have around 12TB in Letbox. Im still in the process of planning to backup such size.

  • @yoursunny said: For my website, the source of truth is the git repository

    Then you travel to Iran or Cuba and Github terminates your account :)

  • I once fucked up something like this, about 1 month old client website. I give him additional 1 year of service.

    Thanked by 1Ympker
  • @William said:

    @yoursunny said: For my website, the source of truth is the git repository

    Then you travel to Iran or Cuba and Github terminates your account :)

    GitHub is now available in Iran btw - https://github.blog/2021-01-05-advancing-developer-freedom-github-is-fully-available-in-iran/

  • yoursunnyyoursunny Member, IPv6 Advocate

    @William said:

    @yoursunny said: For my website, the source of truth is the git repository

    Then you travel to Iran or Cuba and Github terminates your account :)

    1. My website repositories are on Bitbucket not GitHub. They are public repositories but I want them to have less visibility.
    2. I don't travel to evil countries.
  • deankdeank Member, Troll

    Have you considered suing?

    It's the only way to let the world know that how serious you are.

    Thanked by 1yoursunny
  • yoursunnyyoursunny Member, IPv6 Advocate

    @deank said:
    Have you considered suing?

    It's the only way to let the world know that how serious you are.

    Small claims court or lawyer court?

  • @yoursunny said: My website repositories are on Bitbucket

    Bitbucket is under the same sanctions as Github.

    @yoursunny said: I don't travel to evil countries.

    Yea, now it is Cuba and Iran, but before this list also included Russia and Belarus, the US can (and does) randomly add and remove sanctions all the time.

    I had cases of accounts being blocked because the IPv4 space was sold by ICT in Iran to an EU DSL provider, so that also happens...

    Point being, don't rely on US companies.

  • That is a bad PR, for a company that sell back-up storage space. in this case Letbox fucked up big time!

  • I still have to see @key900 's side on this though...

    Thanked by 1ErawanArifNugroho
  • @yokowasis said: I give him additional 1 year of service.

    Something like this would be a good outcome. 1 Month of free service isn't nearly enough. Client shouldn't suffer because provider couldn't give a rats arse about staging environment.

  • jon617jon617 Veteran
    edited January 2021

    Sorry to hear about your data loss. Thanks for letting us know.

    Last week I actually had signed up for a block storage server with letbox for 6 months. Will probably not renew it.

  • @cazrz said:
    Not good.

    Modulesgarden should have worked and test things on staging environment and not on production server. Why the hell will you test the terminate feature on a production server/site.

    Anyway shit happens, just like when the engineer of AWS is playing with the playbook which is actually connected on the production block storage. They also deleted some data and no way to put back. IIRC

    Now, My question is. When using block storage in Letbox and this kind of shit happens again, does it mean that the data in block storage also gets automatically deleted when the account of server gets terminated?

    IIRC, the data is preserved when you reinstall the server using ISO and not template. When you reinstall server using template then the block storage data gets deleted. I haven't tried the terminate server process yet.

    I think I have around 12TB in Letbox. Im still in the process of planning to backup such size.

    I think we're the only ones who read the OP and understand what happened.

    LetBox was letting the developer of ModulesGarden investigate a bug involving ModulesGarden. This wasn't some third party dev working on unknown system for first time. ModulesGarden should be professional and collecting data, not fucking with active accounts. LetBox shouldn't have to explicitly say, "don't fucking delete active customers", it's expected.

    On top of that, they didn't come clean until LetBox questioned them rather than immediately notifying LetBox and possibly having a chance to undo the damage.

    Tl;dr. Fucking ModulesGarden fucking LetBox and customer in the ass. Fucked cociu long enough and now LetBox.

    If this was more than one client that got deleted, I would actually tell ModulesGarden owner they need to fire that dev and remind them they are responsible for properly training their techs. And then promptly investigate replacing ModulesGarden.

    Thanked by 1Chronic
  • @jon617 said:
    Sorry to hear about your data loss. Thanks for letting us know.

    Last week I actually had signed up for a block storage server with letbox for 6 months. Will probably not renew it.

    This is bad logic. This is valuable experience gained.

    Have people not heard the story where a new hire fucks up badly and expects to be fired but the boss surprises them by telling them that they just gained experience and firing them will just result in the next new hire to repeat the same mistake?

    It's things like this where a company says, "fuck that, never again!" And actually come out better in the end.

    1-2 years ago LetBox had a couple of power and network issues over a few months. Then they learned from that and upgraded the power and network and those problems are a thing of the past.

    I'm 1000% certain the next person working on LetBox system will be told 3+ times about that fucking asshole at ModulesGarden and put the fear into them making sure not to do any amateur mistakes.

  • John_MJohn_M Member, Patron Provider

    wow, that's so unfortunate for you. Hopefully you at least have some backup data.

  • @TimboJones said: Have people not heard the story where a new hire fucks up badly and expects to be fired but the boss surprises them by telling them that they just gained experience and firing them will just result in the next new hire to repeat the same mistake?

    @TimboJones said: It's things like this where a company says, "fuck that, never again!" And actually come out better in the end.

    The most notable example of this imo is CloudFlare. Someone accidentally pushed updated WAF rules that lead to a massive spike in CPU usage and took the entire platform down

    The CTO's response was to accept the apology from the dev, tell them not to worry about it and instead focus on better automated testing

    Anyone can make accidents at any time, shit happens

    Thanked by 2yoursunny TimboJones
Sign In or Register to comment.