Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Smallweb LON shared hosting, boom!!!
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Smallweb LON shared hosting, boom!!!

Dear friend,

At 6:03PM UK time on 25th July 2021 I received an email stating that the server and licenses for London were suspended.

Shortly after, I was alerted that the service was infact terminated in error by the infrastructure provider's WHMCS installation as opposed to being suspended.

Because of this, I am sorry to tell you that catastrophic data loss has occurred in London. With the server terminated, no live data remains.

An emergency re-installation of DirectAdmin on a new server began at 7:46PM once the license was re-activated and the server is back online in a "skeleton" mode which currently does not include the likes of CloudLinux and Softaculous.

I am working non-stop to secure and restore any of our available backups. If you have your own backup, please open a ticket and I can re-create your account.

I am beyond sorry for this damaging event.

I will be communicating with you further as I gather more information to provide.

Michael,

SmallWeb.net

Except for my backup months ago, all files are gone.
BOOM!

Comments

  • Its okay you can still get your files..

  • brueggusbrueggus Member, IPv6 Advocate
  • ezethezeth Member, Patron Provider
    edited July 2021

    When I read the title I thought he was talking about me as well :D
    Thankfully he's not haha

    Thanked by 1yoursunny
  • NeoonNeoon Community Contributor, Veteran

    So, WHCMS went Taliban, does that mean, it has a bug, that may exterminate other stuff too, should I be worried?

    Rebell of the Machines? Already this Year?

  • vyas11vyas11 Member

    Backup months ago? That is bigger issue looks like

    Thanked by 1ViridWeb
  • @SmallWeb All the best mate

    Thanked by 1MichaelCee
  • yoursunnyyoursunny Member, IPv6 Advocate

    @Neoon said:
    WHCMS may exterminate other stuff too, should I be worried?

    WHMCS automated involucration, how nice it is?

    P.S. I have @VirMach freebie that will auto-involucrate after 120 days.
    Yet I still run production stuff on there, because my app is fully redundant.

  • Yikes. This is the second time SmallWeb has had catastrophic data loss in a year (LAX1 was the other iirc), and I can't recommend them to anyone else even though they're otherwise a prem service.

  • niceboyniceboy Veteran

    Yesterday, I was thinking of having a service with them! Thankfully, I did not.

  • JamesFJamesF Member, Host Rep

    I don’t get this……. Directadmin has a simple backup that gives offsite backups daily…..

    I am also not sure how WHMCS kills a server….

  • YmpkerYmpker Member
    edited July 2021

    All the best to @SmallWeb . Hope this gets resolved soon. When something like this happens to super-budget hosts (like many are on LE) I always wonder if the trouble was worth it for the pennies they charge.
    Good luck with the restore, mate!

    Thanked by 2MichaelCee afdsafs
  • mmuyskensmmuyskens Member, Host Rep

    @drunkendog said:
    Yikes. This is the second time SmallWeb has had catastrophic data loss in a year (LAX1 was the other iirc), and I can't recommend them to anyone else even though they're otherwise a prem service.

    I thought initially this all sounded a bit sus, but the upstream provider is indeed taking blame for the issue. Just FYI - I confirmed the initial statement is true and this wasn't an issue caused by SmallWeb themselves.

  • @mmuyskens said:
    I thought initially this all sounded a bit sus, but the upstream provider is indeed taking blame for the issue. Just FYI - I confirmed the initial statement is true and this wasn't an issue caused by SmallWeb themselves.

    Thank you, I appreciate that but please don't believe that I am guilt free. As the "driver of the car", I have a responsibility to ensure things like this don't happen, and if they do, that they are resolved completely and although I did not flick the off switch I am the one responsible for flicking it back on.

    @Ympker said:
    All the best to @SmallWeb . Hope this gets resolved soon. When something like this happens to super-budget hosts (like many are on LE) I always wonder if the trouble was worth it for the pennies they charge.
    Good luck with the restore, mate!

    Thanks mate, you're a star.

    @Neoon said:
    So, WHCMS went Taliban, does that mean, it has a bug, that may exterminate other stuff too, should I be worried?

    Rebell of the Machines? Already this Year?

    I say with 99% certainty you should not be concerned in your general use of WHMCS (In this regard)

    @caracal said:
    @SmallWeb All the best mate

    Cheers mate


    I am used to going to bed at 9AM, but these last two nights it's not been out of choice. I'm exhausted, but this is about the customers and not me. So with that said, I will paste the 2nd update email below:

    Dear friend,

    This is a follow up regarding the critical incident that occured in London on 25th July 2021.

    So far, approximately 91 user accounts have been restored from a mix of older emergency backups and customer made backups.

    I estimate this covers approximately 70-80% of the users that existed prior to the data loss.

    Unfortunately, 4 of the user accounts created in July have been ruled out as recoverable and have been recreated.

    Data recovery attempts are still ongoing however I am very sorry to say that the more time passes, the less likely it is that additional data will be recovered.

    If your account(s) have not been restored you are advised to utilise any self-made backups you may have.

    You can open a ticket to request your account is recreated.

    I will continue to apologise for this catastrophic event and for the beyond inconvenience it may cause you. Incidents like this should never happen.

    Michael,
    SmallWeb.net

    Thanked by 1Ympker
  • _MS__MS_ Member

    Who's the Upstream Provider?

  • ericlsericls Member, Patron Provider

    Do they have a back door to delete your data?

  • PaxPax Member
    edited July 2021

    Michael is a good guy, but this was unprofessional.

    I've lost all of my websites and current data (20+ projects). There was an e-mail send out issue I was facing, it was fixed but I was told Michael is running his own dedicated server with DA instance in this location. I fail to understand how a dedicated server gets deleted via WHMCS to the point of full data wipe out, especially since this location "should not be "a re-seller" instance on his side. It's rare to see providers using WHMCS for dedicated servers and having such level of automation.

    If that really is the case, you should seriously switch to another dedi provider, but I feel like it isn't the whole story. You should also inform us who is the provider.

    For me, there was nothing to restore the backups from. While I have some manual backups, they aren't all up to date, but a full dedi termination is the last thing I find possible in this story.

    I seriously hope you re-think the strategy, your upstream providers and put automated backups in place from an admin level. It doesn't seem that backups you have (where applicable), are within the relevant scope (over a month old) as well. So those who got recoveries, got recovered to quite outdated versions. Might be wrong on this, but since there isn't my backup available, I doubt those are recent at all.

    I am a long term customer, and wasn't one of those July accounts,yet full data loss occured.

    Nonetheless, I wish Michael luck to recover from this, from personal perspective, I hope he can put this case to rest and move on, from business perspective, this shouldn't even be a possibility.

    Thanked by 2drunkendog Ympker
  • YmpkerYmpker Member
    edited July 2021

    @Pax said:
    Michael is a good guy, but this was unprofessional.

    I've lost all of my websites and current data (20+ projects). There was an e-mail send out issue I was facing, it was fixed but I was told Michael is running his own dedicated server with DA instance in this location. I fail to understand how a dedicated server gets deleted via WHMCS to the point of full data wipe out, especially since this location "should not be "a re-seller" instance on his side. It's rare to see providers using WHMCS for dedicated servers and having such level of automation.

    If that really is the case, you should seriously switch to another dedi provider, but I feel like it isn't the whole story. You should also inform us who is the provider.

    For me, there was nothing to restore the backups from. While I have some manual backups, they aren't all up to date, but a full dedi termination is the last thing I find possible in this story.

    I seriously hope you re-think the strategy, your upstream providers and put automated backups in place from an admin level. It doesn't seem that backups you have (where applicable), are within the relevant scope (over a month old) as well. So those who got recoveries, got recovered to quite outdated versions. Might be wrong on this, but since there isn't my backup available, I doubt those are recent at all.

    I am a long term customer, and wasn't one of those July accounts,yet full data loss occured.

    Nonetheless, I wish Michael luck to recover from this, from personal perspective, I hope he can put this case to rest and move on, from business perspective, this shouldn't even be a possibility.

    I hear you. This is why I mentioned earlier that it is especially during events like this where I wonder whether the trouble is worth the few pennies Michael or similar LE providers earn from these offers. I agree a lot went wrong and while your comment has been reasonable and not emotionally driven, often enough I see people expecting diamonds for pennies. While it is true that the customer should be able to expect to get what they pay for as outlined by the product description (in this case daily backups), I feel offering backups as part of the product often backfires for small hosts.

    4.5 The contents placed by the customer in the storage media provided by Provider are to be backed up at regular intervals by the customer on his own storage media, which are not those of Provider (backup obligation). The customer is also required to back up his other data independently.

    This extraction from a german provider's ToS has been automatically translated with DeepL. The Provider I am referring to is what I would consider a premium shared hosting provider in Germany and probably the go-to solution I'd recommend people without a doubt. They charge significantly more than @SmallWeb and yet they do still obligate their clients to take their own backups waiving any responsibility. Now, with this certain provider I would not doubt for a second that they'd have disaster recovery backups in place to restore whole servers and/or specific accounts, even though they are not obligated to. With them you also wont get access to the backups they take without contacting support (no Jetbackups or similar). What I am saying is that Clients should be able to expect to get what they paid for but also not expect too much from budget hosts. To Michael, perhaps removing daily backups could save some trouble.

  • Guys,
    We were the upstream provider. Without getting much into details or violating our customers privacy, we had some billing agreement setup with the client and some service extensions.

    Problem was, a lot of the communications were done over discord chat and not helpdesk. Part of me lost track as I have personally been busy with many things (as you may have seen based on my forum interactions or the lack of it lately).

    Unfortunately a wrong flag was placed and that resulted in service getting termed since whmcs didn't update the service renewal but chose to simply termination.

    This was a mutual error. Neither Michale nor I saw it coming. This is also the main reason why I don't offer discord/dm support since things can get lost easily.

    None the less,on our end, we have learnt from the mistake and some of the things will require double approval moving onwards. One from me and one from Roy.

    We apologies to everyone for the mishap.

  • YmpkerYmpker Member

    @seriesn said:
    Guys,
    We were the upstream provider. Without getting much into details or violating our customers privacy, we had some billing agreement setup with the client and some service extensions.

    Problem was, a lot of the communications were done over discord chat and not helpdesk. Part of me lost track as I have personally been busy with many things (as you may have seen based on my forum interactions or the lack of it lately).

    Unfortunately a wrong flag was placed and that resulted in service getting termed since whmcs didn't update the service renewal but chose to simply termination.

    This was a mutual error. Neither Michale nor I saw it coming. This is also the main reason why I don't offer discord/dm support since things can get lost easily.

    None the less,on our end, we have learnt from the mistake and some of the things will require double approval moving onwards. One from me and one from Roy.

    We apologies to everyone for the mishap.

    Props for the transparency and coming out yourself to say this rather than putting Michael in the position to spill the name. Mistakes were made, shit happens. Wishing you guys all the best and I am sure everyone has learnt something from this, at the end of the day.

  • yoursunnyyoursunny Member, IPv6 Advocate

    @Pax said:
    I've lost all of my websites and current data (20+ projects).
    For me, there was nothing to restore the backups from. While I have some manual backups, they aren't all up to date

    It's your fault for failing to secure a disaster recovery plan, with either up-to-date off-site backups or a method to quickly re-deploy your website to another server.
    The provider can burn down the data center or deadpool OfflineServers style at any time, without any advance warning, and you must be prepared for that.

    Thanked by 1chocolateshirt
  • I setup cron job to backup my site weekly, if the provider got something like this, all data gone, I will deploy my backup to other service immediately and never use them anymore. So far, all my provider is good, never got problem. :)

  • PaxPax Member
    edited July 2021

    @yoursunny said:

    @Pax said:
    I've lost all of my websites and current data (20+ projects).
    For me, there was nothing to restore the backups from. While I have some manual backups, they aren't all up to date

    It's your fault for failing to secure a disaster recovery plan, with either up-to-date off-site backups or a method to quickly re-deploy your website to another server.
    The provider can burn down the data center or deadpool OfflineServers style at any time, without any advance warning, and you must be prepared for that.

    I don't agree with your view, but you are entitled to it as much as I am entitled to mine. I put my trust into the provider to provide the service and stability, so I do not need to worry about it myself, which is why I don"t host it locally. In this specific case I cannot fault myself as a buyer, because this was miscommunication on the provider's side and could easily be avoided.

    Refering to OVH, I don't see the correlation. OVH was able to recover a lot of data from customers disks and re-deploy dedi's as well, although the circumstances were much harsher, involving fire.

    Your ideology is that the customer is at fault for not taking daily backups. I partially agree, that clients are responsible for their backups (I took my portion of the blame for not taking daily backups btw), but in my opinion, so are providers at providing stable services. I'd say generally the ratio is at least 70/30, 70 being on the provider, 30 on the client, and negligence on the provider side, generally causes a bigger problem.

    We can go back and forth in discussion, but I believe we won't reach a common ground in our views. I've stayed as transparent as possible,this will be my last comment on this matter as well, because my opinion at the end of the day doesn't do any good for each parties ( me or the provider) :)

    Thanked by 1webclouddev
  • HarambeHarambe Member, Host Rep

    @Pax said: I put my trust into the provider to provide the service and stability, so I do not need to worry about it myself

    You're welcome to take this approach and will probably end up fine most of the time, but just know that there's a very high risk of you getting burned again.

    This is just a warning from someone who may've said the same thing a few years back.

  • @Pax said:
    Your ideology is that the customer is at fault for not taking daily backups. I partially agree, that clients are responsible for their backups (I took my portion of the blame for not taking daily backups btw), but in my opinion, so are providers at providing stable services.

    So you're actually blaming a Low end provider that doesn't guarantee any user data backups whatsoever for...

    checks notes

    ...not having backups?

    Now I know that @SmallWeb has some backups, but it's not weekly or monthly backups as you might expect from mainstream providers. And these providers even put this in their USP's.

    The server going down is on the provider. But not having recent backups while hosting with a provider that doesn't offer that is totally on you. Ideology or not.

  • An update for affected customers:

    Dear friend,

    This is likely the final follow up email regarding the critical incident that occured in London on 25th July 2021. I will of course be communicating directly to those who are facing continuing issues or need assistance.

    The previous update stated that approximately 91 of the affected users had been restored. Upon additional data recovery attempts, I can now say approximately 138 affected users are now restored.

    On top of the 4 users deemed unrecoverable I sincerely regret to announce an additional 21 user accounts are unable to be restored.

    If you are one of those users, I am extremely sorry that your data was lost and cannot be restored. Your account will be recreated so you can utilise any user-made backups you may have. You may open a support ticket to expedite this process.

    Michael,
    SmallWeb.net

    Thanked by 1Ympker
  • To clarify comments made by @debaser and @Pax

    Backups are advertised, but the Terms of Service state they include no warranty or guarantee of their date, accuracy, and integrity. Does this make one person right and one person wrong? No. It is much easier to throw blame at a customer for not taking backups on a thread about a provider that isn’t myself. But being behind the reality, I find it extremely hard to place blame on the customers, who were just casually going about their life until this event occurred. There is responsibility that is on my end and I would like to apologise to all of those involved.

    Looking at this incident, I believe I have failed in the following ways:

    • Ensuring requests and confirmations are logged in the infrastructure provider's database rather than off-site through IM, especially when they are very busy and when the keeping of requests logged for verification is so important for the integrity of services.
    • Verifying the services continued availability when this could have been avoided.
    • Communicating with customers in a timely manner (It took roughly an hour and a half to have the server back online in a skeleton form, but I completely messed up Blesta's SMTP after sending only 19 emails and they were resent approximately 9 hours later then scheduled at around 7AM).
    • Ensuring adequate backups were available. The users I have restored were recovered from a corrupted hard drive which was used for non-primary emergency backups, meaning they are 3-6 weeks out of date and unfortunately due to the corruption several backups contained zero data.

    I will continue to be regretful for this incidence throughout my life and if I can do anything for any affected users then please reach out and I will be there.

Sign In or Register to comment.