Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Advertise on LowEndTalk.com
Letbox terminated client's server by mistake and lose all data
New on LowEndTalk? Please Register and read our Community Rules.

Letbox terminated client's server by mistake and lose all data

lightforcelightforce Member
edited January 14 in General

Jesus, great work for their 9 anniversary. I'm unfamiliar with whmcs, could that be happened?
They claimed they outsourced some exams from some team. And credit for one more month for this case.
SO BE CAREFUL with YOUR OFFICIAL PROJECT, this is not a honoured provider

Thanked by 1JasonM
«1

Comments

  • Adam1Adam1 Member

    So, they are offering no solution to restore at all? they deleted - didnt suspend? thats one fucked up procedure, theres no excusing that at all.

    In my mind, financial compensation is the only remedy here, but thats unlikely.

    I would move away from them, yesterday.

    oh, and backups ofc.

    Thanked by 1Chronic
  • @lightforce I think examination means while checking / inspection / testing / validating / verfying or something like that.

    Atleast in the context of the thread it sounds like that, it cannot be a test ( Like a school test ).

  • thedpthedp Member

    It's simple - sh*t happens.

    Tech/Hosting & Miscellaneous Domain Names For Sale!

  • @lightforce said: SO BE CAREFUL with YOUR OFFICIAL PROJECT

    OFFICIAL projects should have MULTIPLE OFFICIAL backups.

  • TravTrav Member

    This is why you should always back up your data, if it's important enough. Never know when mistakes like this will happen.

    The provider should of had some sort of retention policy in place, like when the VM service is terminated wait 3 days before deleting the data permanently. I've heard of providers doing this just in case something like this happens.

  • Probably million tabs was opened in browser and he pushed delete in the wrong one. It happens. But there is no backups at provider side?!

  • I did back my database, due to the server had lots of 4k tiny files, backup will utilize much resources, so I backuped manually on schedule. Now the site could not be retained because DB and site file snap is not the same time.

    @Trav said:
    This is why you should always back up your data, if it's important enough. Never know when mistakes like this will happen.

    The provider should of had some sort of retention policy in place, like when the VM service is terminated wait 3 days before deleting the data permanently. I've heard of providers doing this just in case something like this happens.

  • Just one more month for compensation. I have not yet made decision as refund or continue to use.

    @Adam1 said:
    So, they are offering no solution to restore at all? they deleted - didnt suspend? thats one fucked up procedure, theres no excusing that at all.

    In my mind, financial compensation is the only remedy here, but thats unlikely.

    I would move away from them, yesterday.

    oh, and backups ofc.

  • @lightforce said:
    Now the site could not be retained because DB and site file snap is not the same time.

    The lesson is: have only one source of truth, and keep it in an atomic unit.

    For my website, the source of truth is the git repository. There's no database, and the PHP scripts are not allowed to create files other than in an ephemeral cache folder that is lost upon reboot. All the content is committed to the git repositories and then rsync'ed to the server.
    Having adopted this system, a recovery takes less than an hour without provider's help.

    https://yoursunny.com/t/2017/yoursunny-com-git/

    Thanked by 2nyamenk ariq01
  • @lightforce said:
    Just one more month for compensation. I have not yet made decision as refund or continue to use.

    @Adam1 said:
    So, they are offering no solution to restore at all? they deleted - didnt suspend? thats one fucked up procedure, theres no excusing that at all.

    In my mind, financial compensation is the only remedy here, but thats unlikely.

    I would move away from them, yesterday.

    oh, and backups ofc.

    I don't find that would be a good compensation for a month caused all data lose. i will never accept this offer.

  • @snt said:
    this guy will promote his website in every possible thread he comment in

    He's doing push-ups for that. And a push-up this year is a new black.

    Thanked by 1yoursunny
  • sntsnt Member

    @LTniger said:

    @snt said:
    this guy will promote his website in every possible thread he comment in

    He's doing push-ups for that. And a push-up this year is a new black.

    he is just full of himself thats all.. and im sure other people here agree to this

  • @snt said:

    @LTniger said:

    @snt said:
    this guy will promote his website in every possible thread he comment in

    He's doing push-ups for that. And a push-up this year is a new black.

    he is just full of himself thats all.. and im sure other people here agree to this

    sunny is cool dude. Chill, community is alright.

  • cazrzcazrz Member
    edited January 14

    Not good.

    Modulesgarden should have worked and test things on staging environment and not on production server. Why the hell will you test the terminate feature on a production server/site.

    Anyway shit happens, just like when the engineer of AWS is playing with the playbook which is actually connected on the production block storage. They also deleted some data and no way to put back. IIRC

    Now, My question is. When using block storage in Letbox and this kind of shit happens again, does it mean that the data in block storage also gets automatically deleted when the account of server gets terminated?

    IIRC, the data is preserved when you reinstall the server using ISO and not template. When you reinstall server using template then the block storage data gets deleted. I haven't tried the terminate server process yet.

    I think I have around 12TB in Letbox. Im still in the process of planning to backup such size.

  • WilliamWilliam Member, Provider

    @yoursunny said: For my website, the source of truth is the git repository

    Then you travel to Iran or Cuba and Github terminates your account :)

  • I once fucked up something like this, about 1 month old client website. I give him additional 1 year of service.

    Thanked by 1Ympker
  • @William said:

    @yoursunny said: For my website, the source of truth is the git repository

    Then you travel to Iran or Cuba and Github terminates your account :)

    GitHub is now available in Iran btw - https://github.blog/2021-01-05-advancing-developer-freedom-github-is-fully-available-in-iran/

  • @William said:

    @yoursunny said: For my website, the source of truth is the git repository

    Then you travel to Iran or Cuba and Github terminates your account :)

    1. My website repositories are on Bitbucket not GitHub. They are public repositories but I want them to have less visibility.
    2. I don't travel to evil countries.
  • deankdeank Member, Troll

    Have you considered suing?

    It's the only way to let the world know that how serious you are.

    Thanked by 1yoursunny

    Have you sued your host yet? Do it now.

  • @deank said:
    Have you considered suing?

    It's the only way to let the world know that how serious you are.

    Small claims court or lawyer court?

  • WilliamWilliam Member, Provider

    @yoursunny said: My website repositories are on Bitbucket

    Bitbucket is under the same sanctions as Github.

    @yoursunny said: I don't travel to evil countries.

    Yea, now it is Cuba and Iran, but before this list also included Russia and Belarus, the US can (and does) randomly add and remove sanctions all the time.

    I had cases of accounts being blocked because the IPv4 space was sold by ICT in Iran to an EU DSL provider, so that also happens...

    Point being, don't rely on US companies.

  • That is a bad PR, for a company that sell back-up storage space. in this case Letbox fucked up big time!

  • I still have to see @key900 's side on this though...

    Thanked by 1ErawanArifNugroho
  • @yokowasis said: I give him additional 1 year of service.

    Something like this would be a good outcome. 1 Month of free service isn't nearly enough. Client shouldn't suffer because provider couldn't give a rats arse about staging environment.

    I repeat, RAID is not backup | Looking for a developer for your next project? - Hire me

  • jon617jon617 Member
    edited January 15

    Sorry to hear about your data loss. Thanks for letting us know.

    Last week I actually had signed up for a block storage server with letbox for 6 months. Will probably not renew it.

  • @cazrz said:
    Not good.

    Modulesgarden should have worked and test things on staging environment and not on production server. Why the hell will you test the terminate feature on a production server/site.

    Anyway shit happens, just like when the engineer of AWS is playing with the playbook which is actually connected on the production block storage. They also deleted some data and no way to put back. IIRC

    Now, My question is. When using block storage in Letbox and this kind of shit happens again, does it mean that the data in block storage also gets automatically deleted when the account of server gets terminated?

    IIRC, the data is preserved when you reinstall the server using ISO and not template. When you reinstall server using template then the block storage data gets deleted. I haven't tried the terminate server process yet.

    I think I have around 12TB in Letbox. Im still in the process of planning to backup such size.

    I think we're the only ones who read the OP and understand what happened.

    LetBox was letting the developer of ModulesGarden investigate a bug involving ModulesGarden. This wasn't some third party dev working on unknown system for first time. ModulesGarden should be professional and collecting data, not fucking with active accounts. LetBox shouldn't have to explicitly say, "don't fucking delete active customers", it's expected.

    On top of that, they didn't come clean until LetBox questioned them rather than immediately notifying LetBox and possibly having a chance to undo the damage.

    Tl;dr. Fucking ModulesGarden fucking LetBox and customer in the ass. Fucked cociu long enough and now LetBox.

    If this was more than one client that got deleted, I would actually tell ModulesGarden owner they need to fire that dev and remind them they are responsible for properly training their techs. And then promptly investigate replacing ModulesGarden.

    Thanked by 1Chronic
  • @jon617 said:
    Sorry to hear about your data loss. Thanks for letting us know.

    Last week I actually had signed up for a block storage server with letbox for 6 months. Will probably not renew it.

    This is bad logic. This is valuable experience gained.

    Have people not heard the story where a new hire fucks up badly and expects to be fired but the boss surprises them by telling them that they just gained experience and firing them will just result in the next new hire to repeat the same mistake?

    It's things like this where a company says, "fuck that, never again!" And actually come out better in the end.

    1-2 years ago LetBox had a couple of power and network issues over a few months. Then they learned from that and upgraded the power and network and those problems are a thing of the past.

    I'm 1000% certain the next person working on LetBox system will be told 3+ times about that fucking asshole at ModulesGarden and put the fear into them making sure not to do any amateur mistakes.

  • John_MJohn_M Member, Provider

    wow, that's so unfortunate for you. Hopefully you at least have some backup data.

    QuadraNet-Data Center Ingenuity Made Simple
    Dedicated Servers - Colocations - Public/Private Cloud - Backup Solutions and More
    Email: Sales[at]QuadraNet.com Sky: John-QuadraNet

  • @TimboJones said: Have people not heard the story where a new hire fucks up badly and expects to be fired but the boss surprises them by telling them that they just gained experience and firing them will just result in the next new hire to repeat the same mistake?

    @TimboJones said: It's things like this where a company says, "fuck that, never again!" And actually come out better in the end.

    The most notable example of this imo is CloudFlare. Someone accidentally pushed updated WAF rules that lead to a massive spike in CPU usage and took the entire platform down

    The CTO's response was to accept the apology from the dev, tell them not to worry about it and instead focus on better automated testing

    Anyone can make accidents at any time, shit happens

    Thanked by 2yoursunny TimboJones
  • It's really bad for you. But it may be a single mistake, which is not a systematic bug. This happened to me one time, at Vultr - a top world provider. Then, they credited me $50 for it.

    After that incident, I always keep backups.

  • Darn... The support techs are terrible... Arent they required to speak atleast simple unbroken English?

  • @gwnd1989 said:
    Darn... The support techs are terrible... Arent they required to speak atleast simple unbroken English?

    No. They required to reply with answered question and/or resolved problem resolution. 0 f given if english is broken. Look at cociu.

  • @snt said:
    this guy will promote his website in every possible thread he comment in

    İ hate this type of answers. Yes you should always do backups. But still it doesn't justify what provider did.

    "Provider is not guilty. You should have do backups"

  • @key900 has my support here, but I definitely understand how that would be very shitty for the customer/OP.

    I noticed a refound was offered by letbox which imo is a very fair gesture of goodwill on an unmanaged service without an enterprise SLA or any liability for data.

    I noticed key900 also mentioned that moving forward more care will be taken, so it seems like the issue was acknowledged, a solution found (to be more careful) and a gesture of goodwill offered in the form of a refund.

    Agreed 100% it's a shitty situation and in this case both parties got screwed in the process.

    I <3 Nexus Bytes and Direct Admin <3

  • @LTniger said:

    @gwnd1989 said:
    Darn... The support techs are terrible... Arent they required to speak atleast simple unbroken English?

    No. They required to reply with answered question and/or resolved problem resolution. 0 f given if english is broken. Look at cociu.

    Well but not every customer thinks the same way... Broken Communication = Broken Sevices to some.

  • jsgjsg Member

    Modulesgarden? Translates to problems and crap. Seen more than enough.

    Letbox is the hoster and hence responsible. Responsible for the software they use. Responsible for their hosting. And responsible for whom they let in to work on their stuff.
    The fact that the Letbox guy didn't respond yet although tagged doesn't help either.

    OTOH @OP came away with relatively little and hopefully repairable damage. One well known provider here (HostSolutions) almost was destroyed by Modulesgarden crap.

    Thanks no.

  • key900key900 Member, Provider
    edited January 15

    Hello,

    Let's me apology for what happened and my delay to respond here, i have been in vacation i don't following up. we have been served a lot of clients for many years and we shouldn't do such issue like that.
    let me example what is happened, i'm not defend or blame since we who responsible for this data. what i collected.

    1 - There was 2 issue in one ticket submitted to modulesgarden regarding IPS management.
    2 - 2 x account provided one test account and the OP account since it effected with this bug as example.
    3 - By mistake the agent of modulegarden confused the terminated the OP instead of the testing account.
    4 - ModuleGarden not notification us regards the account until the OP open with us a ticket.
    5- YES THEY did that mistake as we are and I apology for that, We will do our best to solve This issue, continue the good cooperation.
    6- WE directly have conversation with the modulegarden manager to make a detailed internal audit on this matter and make sure that this will never happen again.
    7- While i checked the OP account i can see that the last payment where made from the free credit we gives to all our client in BlackFriday so i want to be flexible as possible and Yes we made this mistake and would ask how the client need as compensation .

    Thanks

    *LetBox.Com Make it Simple, Los Angeles & Dallas & New Jersey - Ryzen Pure NVMe + Block Storage Boxes!. Discord Community https://discord.gg/g6dqjmm
  • cazrzcazrz Member

    @key900 isn't that you should have a waiting period mechanism or putting the actual status of the trigger to "on hold" or something for some given time like 1 or 2 days before it actually deletes the data?

    Thanked by 1Trav
  • @key900 said:
    7- While i checked the OP account i can see that the last payment where made from the free credit we gives to all our client in BlackFriday

    Umm, wtf? First I'm hearing of this. Just checked, looks like $15 USD credit on my account!?! How did you communicate this to your customers? I don't recall getting an email.

    Wow, thanks.

  • I remember someone posted something about OVH kicked the guy out so he lost all his 10 years of data. Again, not to say whose fault was that or this incident, backup should be your basic principle of production.

    Good luck.

  • key900key900 Member, Provider
    edited January 15

    @TimboJones said:

    @key900 said:
    7- While i checked the OP account i can see that the last payment where made from the free credit we gives to all our client in BlackFriday

    Umm, wtf? First I'm hearing of this. Just checked, looks like $15 USD credit on my account!?! How did you communicate this to your customers? I don't recall getting an email.

    Wow, thanks.

    If you don't received our mail regarding the free credit in BF then i had to blame mxrouting for their Mail services.

    regards

    *LetBox.Com Make it Simple, Los Angeles & Dallas & New Jersey - Ryzen Pure NVMe + Block Storage Boxes!. Discord Community https://discord.gg/g6dqjmm
  • @TimboJones said:

    @key900 said:
    7- While i checked the OP account i can see that the last payment where made from the free credit we gives to all our client in BlackFriday

    Umm, wtf? First I'm hearing of this. Just checked, looks like $15 USD credit on my account!?! How did you communicate this to your customers? I don't recall getting an email.

    Wow, thanks.

    I received $5....opened ticket about an invoice suddenly marked paid without any payment from me...and they reply that it was a gift from them in form of credit....perhaps you didnt monitor your invoice that often.....

    AhmadShamli
    Get Free USD10 in DigitalOcean credit https://www.digitalocean.com/?refcode=c224f768eb1c
  • This is my last payment, all paid from free credit?
    Gave me one more month as your staff said, I do not accept this condition cause it has nothing to do with free credit or my real payment.

    @key900 said: 7- While i checked the OP account i can see that the last payment where made from the free credit we gives to all our client in BlackFriday so i want to be flexible as possible and Yes we made this mistake and would ask how the client need as compensation .

  • raindog308raindog308 Administrator, Moderator

    @key900 said: let me example what is happened,

    If this had been @VirMach the explanation would have been at least 20,000 words.

    For LET support, please visit the support desk.

  • @Shamli said:

    @TimboJones said:

    @key900 said:
    7- While i checked the OP account i can see that the last payment where made from the free credit we gives to all our client in BlackFriday

    Umm, wtf? First I'm hearing of this. Just checked, looks like $15 USD credit on my account!?! How did you communicate this to your customers? I don't recall getting an email.

    Wow, thanks.

    I received $5....opened ticket about an invoice suddenly marked paid without any payment from me...and they reply that it was a gift from them in form of credit....perhaps you didnt monitor your invoice that often.....

    What? I receive emailed invoices just fine. I just didn't have any between now and black Friday. The fact that I have all $15 credit should have been a clue to you. I'm just saying, didn't get email, didn't see it mentioned on LET and not on their Discourse.

    They totally missed marketing and good will from their credit.

    Thanked by 2key900 skorous
  • @lightforce said:
    This is my last payment, all paid from free credit?
    Gave me one more month as your staff said, I do not accept this condition cause it has nothing to do with free credit or my real payment.

    @key900 said: 7- While i checked the OP account i can see that the last payment where made from the free credit we gives to all our client in BlackFriday so i want to be flexible as possible and Yes we made this mistake and would ask how the client need as compensation .

    Wth? What condition?

    Jesus Christ, what part are you having difficulty with? You only had $1.70 credit left, so you paid $1.60 remainder. This isn't not complex at all and I worry for when you actually get a confusing bill.

    The rest of the credit likely paid for other services in December.

    Thanked by 2key900 AlwaysSkint
  • jarjar Provider
    edited January 16

    @key900 said:
    If you don't received our mail regarding the free credit in BF then i had to blame mxrouting for their Mail services.

    regards

    Always happy to follow up on this. I only see 2 emails since Black Friday from the domain that made it to the end of our stack, meaning they were either continually deferred, rejected for a reason that is suspected to be related to IP (forcing them to retry from other IPs and go through the stack until the end if necessary), or their mail servers were down at the time. Those two were delivered once they reached the end of the stack (spamwall).

    Almost every single time someone says they didn't receive an email, it's either that they use Hotmail and it was accepted and blackholed (so none of us can see anything other than it being delivered successfully), or pebkac. If I bet $100 on every report I'd have a pool of money and I'd invite you over for beer and a rather dry swim party.

    On another note:

    Letbox is actually one of the few customers I've proactively whitelisted on the infrastructure without request, because @key900 handles his shit like a boss.

    Thanked by 1key900
  • key900key900 Member, Provider
    edited January 16

    @jar said:

    @key900 said:
    If you don't received our mail regarding the free credit in BF then i had to blame mxrouting for their Mail services.

    regards

    Always happy to follow up on this. I only see 2 emails since Black Friday from the domain that made it to the end of our stack, meaning they were either continually deferred, rejected for a reason that is suspected to be related to IP (forcing them to retry from other IPs and go through the stack until the end if necessary), or their mail servers were down at the time. Those two were delivered once they reached the end of the stack (spamwall).

    Almost every single time someone says they didn't receive an email, it's either that they use Hotmail and it was accepted and blackholed (so none of us can see anything other than it being delivered successfully), or pebkac. If I bet $100 on every report I'd have a pool of money and I'd invite you over for beer and a rather dry swim party.

    On another note:

    Letbox is actually one of the few customers I've proactively whitelisted on the infrastructure without request, because @key900 handles his shit like a boss.

    This kind of true i saw got whitelisted for while ago and your services really amazing I really appreciate i was just kidding .

    @TimboJones said:

    @lightforce said:
    This is my last payment, all paid from free credit?
    Gave me one more month as your staff said, I do not accept this condition cause it has nothing to do with free credit or my real payment.

    @key900 said: 7- While i checked the OP account i can see that the last payment where made from the free credit we gives to all our client in BlackFriday so i want to be flexible as possible and Yes we made this mistake and would ask how the client need as compensation .

    Wth? What condition?

    Jesus Christ, what part are you having difficulty with? You only had $1.70 credit left, so you paid $1.60 remainder. This isn't not complex at all and I worry for when you actually get a confusing bill.

    The rest of the credit likely paid for other services in December.

    Ya $15 yearly services and $5 for monthly. We announced it in Discord and almost everywhere! I have to mention next BF we will do it again.

    @TimboJones said:

    @lightforce said:

    The credit where spend on other invoice please check all the invoices. And again I’m flexible to fixing this problem with you whatever you feel compensation just PM. If you need time to think about it we shall waiting for you.

    I also noted that you have different services with us since 2015 and I appreciate those years with us. I waiting your PM or i shall do?

    Best regards

    Thanked by 2jar TimboJones
    *LetBox.Com Make it Simple, Los Angeles & Dallas & New Jersey - Ryzen Pure NVMe + Block Storage Boxes!. Discord Community https://discord.gg/g6dqjmm
  • key900key900 Member, Provider

    @raindog308 said:

    @key900 said: let me example what is happened,

    If this had been @VirMach the explanation would have been at least 20,000 words.

    I have to go to straight of the matter and admit the mistakes. Talking a lot does not solve any problems. This is why I would like us to cooperate in solving problems without wasting any time. Mistakes are part of our human nature But this not forgivable we had to learned from what happened and make sure never happened again.

    Best regards

    *LetBox.Com Make it Simple, Los Angeles & Dallas & New Jersey - Ryzen Pure NVMe + Block Storage Boxes!. Discord Community https://discord.gg/g6dqjmm
Sign In or Register to comment.