New on LowEndTalk? Please Register and read our Community Rules.
Letbox terminated client's server by mistake and lose all data
lightforce
Member
Jesus, great work for their 9 anniversary. I'm unfamiliar with whmcs, could that be happened?
They claimed they outsourced some exams from some team. And credit for one more month for this case.
SO BE CAREFUL with YOUR OFFICIAL PROJECT, this is not a honoured provider
Thanked by 1JasonM
LowEndBox & LowEndTalk.
Comments
So, they are offering no solution to restore at all? they deleted - didnt suspend? thats one fucked up procedure, theres no excusing that at all.
In my mind, financial compensation is the only remedy here, but thats unlikely.
I would move away from them, yesterday.
oh, and backups ofc.
@lightforce I think examination means while checking / inspection / testing / validating / verfying or something like that.
Atleast in the context of the thread it sounds like that, it cannot be a test ( Like a school test ).
It's simple - sh*t happens.
Tech/Hosting & Miscellaneous Domain Names For Sale!
OFFICIAL projects should have MULTIPLE OFFICIAL backups.
This is why you should always back up your data, if it's important enough. Never know when mistakes like this will happen.
The provider should of had some sort of retention policy in place, like when the VM service is terminated wait 3 days before deleting the data permanently. I've heard of providers doing this just in case something like this happens.
Probably million tabs was opened in browser and he pushed delete in the wrong one. It happens. But there is no backups at provider side?!
#!/Bashblog.net
I did back my database, due to the server had lots of 4k tiny files, backup will utilize much resources, so I backuped manually on schedule. Now the site could not be retained because DB and site file snap is not the same time.
Just one more month for compensation. I have not yet made decision as refund or continue to use.
The lesson is: have only one source of truth, and keep it in an atomic unit.
For my website, the source of truth is the git repository. There's no database, and the PHP scripts are not allowed to create files other than in an ephemeral cache folder that is lost upon reboot. All the content is committed to the git repositories and then rsync'ed to the server.
Having adopted this system, a recovery takes less than an hour without provider's help.
https://yoursunny.com/t/2017/yoursunny-com-git/
yoursunny.com; stallion coder; push-up specialist; my VPS list; Antarctica summer host; treefiddy
this guy will promote his website in every possible thread he comment in
I don't find that would be a good compensation for a month caused all data lose. i will never accept this offer.
He's doing push-ups for that. And a push-up this year is a new black.
#!/Bashblog.net
he is just full of himself thats all.. and im sure other people here agree to this
sunny is cool dude. Chill, community is alright.
#!/Bashblog.net
Not good.
Modulesgarden should have worked and test things on staging environment and not on production server. Why the hell will you test the terminate feature on a production server/site.
Anyway shit happens, just like when the engineer of AWS is playing with the playbook which is actually connected on the production block storage. They also deleted some data and no way to put back. IIRC
Now, My question is. When using block storage in Letbox and this kind of shit happens again, does it mean that the data in block storage also gets automatically deleted when the account of server gets terminated?
IIRC, the data is preserved when you reinstall the server using ISO and not template. When you reinstall server using template then the block storage data gets deleted. I haven't tried the terminate server process yet.
I think I have around 12TB in Letbox. Im still in the process of planning to backup such size.
Then you travel to Iran or Cuba and Github terminates your account
IP6.IM
I once fucked up something like this, about 1 month old client website. I give him additional 1 year of service.
GitHub is now available in Iran btw - https://github.blog/2021-01-05-advancing-developer-freedom-github-is-fully-available-in-iran/
yoursunny.com; stallion coder; push-up specialist; my VPS list; Antarctica summer host; treefiddy
Have you considered suing?
It's the only way to let the world know that how serious you are.
Have you sued your host yet? Do it now.
Small claims court or lawyer court?
yoursunny.com; stallion coder; push-up specialist; my VPS list; Antarctica summer host; treefiddy
Bitbucket is under the same sanctions as Github.
Yea, now it is Cuba and Iran, but before this list also included Russia and Belarus, the US can (and does) randomly add and remove sanctions all the time.
I had cases of accounts being blocked because the IPv4 space was sold by ICT in Iran to an EU DSL provider, so that also happens...
Point being, don't rely on US companies.
IP6.IM
That is a bad PR, for a company that sell back-up storage space. in this case Letbox fucked up big time!
I still have to see @key900 's side on this though...
Something like this would be a good outcome. 1 Month of free service isn't nearly enough. Client shouldn't suffer because provider couldn't give a rats arse about staging environment.
I repeat, RAID is not backup | Looking for a developer for your next project? - Hire me
Sorry to hear about your data loss. Thanks for letting us know.
Last week I actually had signed up for a block storage server with letbox for 6 months. Will probably not renew it.
I think we're the only ones who read the OP and understand what happened.
LetBox was letting the developer of ModulesGarden investigate a bug involving ModulesGarden. This wasn't some third party dev working on unknown system for first time. ModulesGarden should be professional and collecting data, not fucking with active accounts. LetBox shouldn't have to explicitly say, "don't fucking delete active customers", it's expected.
On top of that, they didn't come clean until LetBox questioned them rather than immediately notifying LetBox and possibly having a chance to undo the damage.
Tl;dr. Fucking ModulesGarden fucking LetBox and customer in the ass. Fucked cociu long enough and now LetBox.
If this was more than one client that got deleted, I would actually tell ModulesGarden owner they need to fire that dev and remind them they are responsible for properly training their techs. And then promptly investigate replacing ModulesGarden.
This is bad logic. This is valuable experience gained.
Have people not heard the story where a new hire fucks up badly and expects to be fired but the boss surprises them by telling them that they just gained experience and firing them will just result in the next new hire to repeat the same mistake?
It's things like this where a company says, "fuck that, never again!" And actually come out better in the end.
1-2 years ago LetBox had a couple of power and network issues over a few months. Then they learned from that and upgraded the power and network and those problems are a thing of the past.
I'm 1000% certain the next person working on LetBox system will be told 3+ times about that fucking asshole at ModulesGarden and put the fear into them making sure not to do any amateur mistakes.
wow, that's so unfortunate for you. Hopefully you at least have some backup data.
QuadraNet-Data Center Ingenuity Made Simple
Dedicated Servers - Colocations - Public/Private Cloud - Backup Solutions and More
Email: Sales[at]QuadraNet.com Sky: John-QuadraNet
The most notable example of this imo is CloudFlare. Someone accidentally pushed updated WAF rules that lead to a massive spike in CPU usage and took the entire platform down
The CTO's response was to accept the apology from the dev, tell them not to worry about it and instead focus on better automated testing
Anyone can make accidents at any time, shit happens
It's really bad for you. But it may be a single mistake, which is not a systematic bug. This happened to me one time, at Vultr - a top world provider. Then, they credited me $50 for it.
After that incident, I always keep backups.
Darn... The support techs are terrible... Arent they required to speak atleast simple unbroken English?
No. They required to reply with answered question and/or resolved problem resolution. 0 f given if english is broken. Look at cociu.
#!/Bashblog.net
İ hate this type of answers. Yes you should always do backups. But still it doesn't justify what provider did.
"Provider is not guilty. You should have do backups"
@key900 has my support here, but I definitely understand how that would be very shitty for the customer/OP.
I noticed a refound was offered by letbox which imo is a very fair gesture of goodwill on an unmanaged service without an enterprise SLA or any liability for data.
I noticed key900 also mentioned that moving forward more care will be taken, so it seems like the issue was acknowledged, a solution found (to be more careful) and a gesture of goodwill offered in the form of a refund.
Agreed 100% it's a shitty situation and in this case both parties got screwed in the process.
I
Nexus Bytes and Direct Admin 
Well but not every customer thinks the same way... Broken Communication = Broken Sevices to some.
Modulesgarden? Translates to problems and crap. Seen more than enough.
Letbox is the hoster and hence responsible. Responsible for the software they use. Responsible for their hosting. And responsible for whom they let in to work on their stuff.
The fact that the Letbox guy didn't respond yet although tagged doesn't help either.
OTOH @OP came away with relatively little and hopefully repairable damage. One well known provider here (HostSolutions) almost was destroyed by Modulesgarden crap.
Thanks no.
Hello,
Let's me apology for what happened and my delay to respond here, i have been in vacation i don't following up. we have been served a lot of clients for many years and we shouldn't do such issue like that.
let me example what is happened, i'm not defend or blame since we who responsible for this data. what i collected.
1 - There was 2 issue in one ticket submitted to modulesgarden regarding IPS management.
2 - 2 x account provided one test account and the OP account since it effected with this bug as example.
3 - By mistake the agent of modulegarden confused the terminated the OP instead of the testing account.
4 - ModuleGarden not notification us regards the account until the OP open with us a ticket.
5- YES THEY did that mistake as we are and I apology for that, We will do our best to solve This issue, continue the good cooperation.
6- WE directly have conversation with the modulegarden manager to make a detailed internal audit on this matter and make sure that this will never happen again.
7- While i checked the OP account i can see that the last payment where made from the free credit we gives to all our client in BlackFriday so i want to be flexible as possible and Yes we made this mistake and would ask how the client need as compensation .
Thanks
@key900 isn't that you should have a waiting period mechanism or putting the actual status of the trigger to "on hold" or something for some given time like 1 or 2 days before it actually deletes the data?
Umm, wtf? First I'm hearing of this. Just checked, looks like $15 USD credit on my account!?! How did you communicate this to your customers? I don't recall getting an email.
Wow, thanks.
I remember someone posted something about OVH kicked the guy out so he lost all his 10 years of data. Again, not to say whose fault was that or this incident, backup should be your basic principle of production.
Good luck.
If you don't received our mail regarding the free credit in BF then i had to blame mxrouting for their Mail services.
regards
I received $5....opened ticket about an invoice suddenly marked paid without any payment from me...and they reply that it was a gift from them in form of credit....perhaps you didnt monitor your invoice that often.....
Get Free USD10 in DigitalOcean credit https://www.digitalocean.com/?refcode=c224f768eb1c
This is my last payment, all paid from free credit?
Gave me one more month as your staff said, I do not accept this condition cause it has nothing to do with free credit or my real payment.
If this had been @VirMach the explanation would have been at least 20,000 words.
For LET support, please visit the support desk.
What? I receive emailed invoices just fine. I just didn't have any between now and black Friday. The fact that I have all $15 credit should have been a clue to you. I'm just saying, didn't get email, didn't see it mentioned on LET and not on their Discourse.
They totally missed marketing and good will from their credit.
Wth? What condition?
Jesus Christ, what part are you having difficulty with? You only had $1.70 credit left, so you paid $1.60 remainder. This isn't not complex at all and I worry for when you actually get a confusing bill.
The rest of the credit likely paid for other services in December.
Always happy to follow up on this. I only see 2 emails since Black Friday from the domain that made it to the end of our stack, meaning they were either continually deferred, rejected for a reason that is suspected to be related to IP (forcing them to retry from other IPs and go through the stack until the end if necessary), or their mail servers were down at the time. Those two were delivered once they reached the end of the stack (spamwall).
Almost every single time someone says they didn't receive an email, it's either that they use Hotmail and it was accepted and blackholed (so none of us can see anything other than it being delivered successfully), or pebkac. If I bet $100 on every report I'd have a pool of money and I'd invite you over for beer and a rather dry swim party.
On another note:
Letbox is actually one of the few customers I've proactively whitelisted on the infrastructure without request, because @key900 handles his shit like a boss.
https://mxroute.blackfriday/
This kind of true i saw got whitelisted for while ago and your services really amazing I really appreciate i was just kidding .
Ya $15 yearly services and $5 for monthly. We announced it in Discord and almost everywhere! I have to mention next BF we will do it again.
The credit where spend on other invoice please check all the invoices. And again I’m flexible to fixing this problem with you whatever you feel compensation just PM. If you need time to think about it we shall waiting for you.
I also noted that you have different services with us since 2015 and I appreciate those years with us. I waiting your PM or i shall do?
Best regards
I have to go to straight of the matter and admit the mistakes. Talking a lot does not solve any problems. This is why I would like us to cooperate in solving problems without wasting any time. Mistakes are part of our human nature But this not forgivable we had to learned from what happened and make sure never happened again.
Best regards