Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Possible Data Leak - HostDoc - Page 2
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Possible Data Leak - HostDoc

2

Comments

  • jarjar Patron Provider, Top Host, Veteran
    edited January 2020

    Found in headers from a curl:

    X-Server-Powered-By: Engintron

    https://engintron.com

    "with an additional micro-cache layer to significantly improve performance for dynamic content generated by CMSs like WordPress, Joomla or Drupal"

    Might this help to identify the cause? This is a significant stack that focuses on caching. Looks like it uses APC + memcached. Could it be caching the dynamic data and returning it to other visitors when they hit the same URLs?

  • RossGRossG Member, Host Rep

    @jar said:
    Found in headers from a curl:

    X-Server-Powered-By: Engintron

    https://engintron.com

    Might this help to identify the cause?

    The cause is (most probably) the micro caching with Engintron:

    if you get 100 visitors requesting the same page in 1 sec, generate the page from the absolute first visitor and then serve the rest 99 visitors the cached copy of that page

    If you curl and look for the "x-nginx-cache-status" header, then quickly curl again within a second you should see it turn from EXPIRED to HIT.

    From what others have posted, I assume this is what happened - somebody logged in then another client was served the cached version of their dashboard.

    Looks like a really simple configuration issue, hopefully it should be able to be resolved quite easily.

    Thanked by 3jar dahartigan FHR
  • FranciscoFrancisco Top Host, Host Rep, Veteran

    jar said: Might this help to identify the cause? This is a significant stack that focuses on caching.

    For sure.

    We tried it on shared but you have to turn off basically all caching if there's cookies just to be safe.

    Francisco

    Thanked by 3jar skorous FHR
  • MikePTMikePT Moderator, Patron Provider, Veteran

    @jar said:
    Found in headers from a curl:

    X-Server-Powered-By: Engintron

    https://engintron.com

    "with an additional micro-cache layer to significantly improve performance for dynamic content generated by CMSs like WordPress, Joomla or Drupal"

    Might this help to identify the cause? This is a significant stack that focuses on caching. Looks like it uses APC + memcached. Could it be caching the dynamic data and returning it to other visitors when they hit the same URLs?

    May very well be it.

    Thanked by 1dahartigan
  • MikeAMikeA Member, Patron Provider
    edited January 2020

    @jar @MikePT When I first started doing cPanel hosting 2-3 years ago I used Engintron, had tons of issues with caching forums that would cause forum users to see others profiles. I would say that's definitely the issue. I think I disabled caching in Enginton completely but eventually dumped it because of some other smaller issues.

  • I gave up on engintron a few years back - just wasn't playing 'nice' with oscommerce stuff (WHM/cPanel VPS).

  • Oh dear! This is sad.
    Hopefully HostDoc knows what the law says. I don't want to see HostDoc going down because of the big fines. This is not a small thing. It needs 100% focus.

  • Is just name, email and address. Honestly this is already leaked around, even in your domain whois... or in some hosting db dump. Nothing really sensitive.

    Chill....

    Thanked by 2jar BlaZe
  • MikeAMikeA Member, Patron Provider

    @Hxxx said:
    Is just name, email and address. Honestly this is already leaked around, even in your domain whois... or in some hosting db dump. Nothing really sensitive.

    Chill....

    Tickets can have sensitive information. With the way caching works many sensitive things can be leaked that aren't just that. But yeah, your basic info is everywhere.

    Thanked by 1dahartigan
  • @jar said:
    Found in headers from a curl:

    X-Server-Powered-By: Engintron

    https://engintron.com

    "with an additional micro-cache layer to significantly improve performance for dynamic content generated by CMSs like WordPress, Joomla or Drupal"

    Might this help to identify the cause? This is a significant stack that focuses on caching. Looks like it uses APC + memcached. Could it be caching the dynamic data and returning it to other visitors when they hit the same URLs?

    Nice find! I would actually be highly surprised if it didn't turn out to be that afterall given that just about everything else has been pinpointed as the cause (tawk, whmcs, cosmic rays etc etc)

    The fact that it is also a very logical and likely explanation for the issue helps too :)

  • @MikeA said:

    @Hxxx said:
    Is just name, email and address. Honestly this is already leaked around, even in your domain whois... or in some hosting db dump. Nothing really sensitive.

    Chill....

    Tickets can have sensitive information. With the way caching works many sensitive things can be leaked that aren't just that. But yeah, your basic info is everywhere.

    Well I mean... yeah if you put your password in the title of a ticket lol. But otherwise based on the info here is just titles.

  • @MikeA said:

    @Hxxx said:
    Is just name, email and address. Honestly this is already leaked around, even in your domain whois... or in some hosting db dump. Nothing really sensitive.

    Chill....

    Tickets can have sensitive information. With the way caching works many sensitive things can be leaked that aren't just that.

    Exactly this. Enough can be gleamed by what's leaking here to successfully use social engineering against the provider.

    But yeah, your basic info is everywhere.

    True, but generally not tied to a service in a way that could be used against you. I could give a stranger who lives on the other side of the country the keys to my house, and as long as he had no idea where I live I'd be safe. Imagine if I gave the keys to my house to someone random on my street..

  • jarjar Patron Provider, Top Host, Veteran

    @MikeA said:

    @Hxxx said:
    Is just name, email and address. Honestly this is already leaked around, even in your domain whois... or in some hosting db dump. Nothing really sensitive.

    Chill....

    Tickets can have sensitive information. With the way caching works many sensitive things can be leaked that aren't just that. But yeah, your basic info is everywhere.

    I think it's safe to assume that only pages without specific IDs in the URL would have been leaked, unless someone started cycling through IDs. At least that can technically, although quite time consuming, be audited to see if any pages with IDs (product ID, ticket ID) were viewed by someone who hadn't logged into the matching account.

    Thanked by 1Clouvider
  • @Rhys said:

    @HostDoc said:

    No, not yet.
    As is evident, the cause was not immediately known. As much as it seems the root cause has now been identified, I am still worried and would like to monitor a while longer before details are submitted.


    The client area has been taken down numerous times for us to carry out work regarding this matter. It was never just left operational while knowing it was leaking.

    As much as many might not like the brand or my responses to threads/toxic comments/tickets, one thing I have always strived to provide is a decent service at the price point.

    So you've known about it for quite some time, and also known that data was leaking during that time yet have failed to report it within the required time defined by the GDPR?

    "At a glance
    The GDPR introduces a duty on all organisations to report certain types of personal data breach to the relevant supervisory authority. You must do this within 72 hours of becoming aware of the breach, where feasible."

    Source: https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/personal-data-breaches/

    Here's an interesting quote from https://hostdoc.co.uk/privacy-policy/

    Specifically, your personal data will be stored in accordance with the Payment Card Industry Data Security Standard

  • MikePTMikePT Moderator, Patron Provider, Veteran

    @MikeA said:
    @jar @MikePT When I first started doing cPanel hosting 2-3 years ago I used Engintron, had tons of issues with caching forums that would cause forum users to see others profiles. I would say that's definitely the issue. I think I disabled caching in Enginton completely but eventually dumped it because of some other smaller issues.

    Never liked that piece of shit. :P

  • RhysRhys Member, Host Rep

    @dahartigan said:

    @Rhys said:

    @HostDoc said:

    No, not yet.
    As is evident, the cause was not immediately known. As much as it seems the root cause has now been identified, I am still worried and would like to monitor a while longer before details are submitted.


    The client area has been taken down numerous times for us to carry out work regarding this matter. It was never just left operational while knowing it was leaking.

    As much as many might not like the brand or my responses to threads/toxic comments/tickets, one thing I have always strived to provide is a decent service at the price point.

    So you've known about it for quite some time, and also known that data was leaking during that time yet have failed to report it within the required time defined by the GDPR?

    "At a glance
    The GDPR introduces a duty on all organisations to report certain types of personal data breach to the relevant supervisory authority. You must do this within 72 hours of becoming aware of the breach, where feasible."

    Source: https://ico.org.uk/for-organisations/guide-to-data-protection/guide-to-the-general-data-protection-regulation-gdpr/personal-data-breaches/

    Here's an interesting quote from https://hostdoc.co.uk/privacy-policy/

    Specifically, your personal data will be stored in accordance with the Payment Card Industry Data Security Standard

    I'd love to see their PCI compliance cert.

    Thanked by 1dahartigan
  • PieHasBeenEatenPieHasBeenEaten Member, Host Rep

    Someone call the nurse the doc is out!

    Thanked by 1TimboJones
  • JordJord Moderator, Host Rep
  • Looks like whmcs is back online and accepting signups and payments again.. does this mean it's fixed? What was the problem?

  • jarjar Patron Provider, Top Host, Veteran

    @dahartigan said:
    Looks like whmcs is back online and accepting signups and payments again.. does this mean it's fixed? What was the problem?

    X-Nginx-Cache-Status: BYPASS

    Cache had to be disabled and it looks to have been.

    Thanked by 3dahartigan uptime DP
  • deankdeank Member, Troll

    When in doubt, C4.

    Blow it up and all problems will be gone.

    Thanked by 3RossG dahartigan FHR
  • DPDP Administrator, The Domain Guy

    This oughta go into some KB

  • Wait does this happen only on the computer where you logged in and logged out? Or your session is cached even after logging out and anyone can see those details not only on the computer you logged in?

  • @dahartigan said:

    @MikeA said:

    @Hxxx said:
    Is just name, email and address. Honestly this is already leaked around, even in your domain whois... or in some hosting db dump. Nothing really sensitive.

    Chill....

    Tickets can have sensitive information. With the way caching works many sensitive things can be leaked that aren't just that.

    Exactly this. Enough can be gleamed by what's leaking here to successfully use social engineering against the provider.

    But yeah, your basic info is everywhere.

    True, but generally not tied to a service in a way that could be used against you. I could give a stranger who lives on the other side of the country the keys to my house, and as long as he had no idea where I live I'd be safe. Imagine if I gave the keys to my house to someone random on my street..

    So they can also see ticket contents not only the title? Yes sometimes there are sensitive info in ticket replies...

  • @timelapse said:
    anyone can see those details not only on the computer you logged in?

    Correct. All someone needs to do is visit the client area and you will see the account and personal details of another customer who could be on the other side of the planet.

    @timelapse said:

    @dahartigan said:

    @MikeA said:

    @Hxxx said:
    Is just name, email and address. Honestly this is already leaked around, even in your domain whois... or in some hosting db dump. Nothing really sensitive.

    Chill....

    Tickets can have sensitive information. With the way caching works many sensitive things can be leaked that aren't just that.

    Exactly this. Enough can be gleamed by what's leaking here to successfully use social engineering against the provider.

    But yeah, your basic info is everywhere.

    True, but generally not tied to a service in a way that could be used against you. I could give a stranger who lives on the other side of the country the keys to my house, and as long as he had no idea where I live I'd be safe. Imagine if I gave the keys to my house to someone random on my street..

    So they can also see ticket contents not only the title? Yes sometimes there are sensitive info in ticket replies...

    Possibly, if they visited a ticket page with a ticket id in the URL, but this is thankfully less likely and requires a bad actor to exploit, unlike the current issue where the system just indiscriminately offers up the details to anyone regardless of their motivation.

  • Further from our previous announcement of a possible data leak, action has now been implemented that will see this issue eradicated moving forward.
    The client area is once again active and clients can now manage their accounts.
    
    
    What was the cause?
    
    With no further evidence of any other application caching the data, it is still believed our tawk.to module was the cause of the leak.
    Micro caching as well as an nginx plugin (Engintron) were also speculated as possible culprits. While these are plausible causes, during earlier troubleshooting of the issue, engintron had previously been disabled as well as global caching. The symptoms of the leak however were still present.
    Removing the tawk.to module saw the symptoms temporarily vanish for a month or so later at which point the tawk.to code was also removed from the footer.tpl.
    
    
    
    What has been done to rectify this and avoid it from happening again?
    
    A rebuild of the client area has been carried out.
    Tawk.to has been removed from the client area.
    Cron job added to clear nginx cache every 24 hours.
    Cron job added to clear template cache every 12 hours.
    Engintron configuration and rules have been tweaked.
    Static caching of client area has been disabled.
    Micro caching of the client dashboard has been disabled.
    Php configuration tweaks.
    
    
    What has been done regarding GDPR and PCI ?
    
    ICO have been contacted and alerted about the data leak by phone.
    An assessment was carried out with information collated about the breach, possible causes and what we intend to do to avoid such leaks again in the future.
    We have been offered advice for future security practices.
    
    As a company utilizing Paypal and Stripe (stripe elements) as our card processors, we ensure card data never touches our servers and is therefore processed securely offsite.
    There is no evidence that payment methods, payments  or available  credit were leaked, however, we are PCI compliant in the way in which we accept card payments.
    
    
    Anything else?
    
    Here at HostDoc, we prescribe hosting medication.
    PR never has been a strong point unfortunately. But please never let that define the service.
    
    I would like to apologize to all HostDoc clients.
    It is never an easy decision to decide to stick with a provider after such an event. It is an even harder decision when the provider lacks the skills to compose themselves in an appropriate manner publicly.
    Security is taken very seriously hence this is not the first time this issue has been addressed and will be continually monitored over the coming months.
    
    Thank you to those who continue to support and utilize HostDoc for their hosting needs.
    
    
    
    
    
    Kind regards
    HostDoc Hosting Team
    
    Thanked by 2uptime vyas11
  • MikePTMikePT Moderator, Patron Provider, Veteran

    Very reasonable reply from Doc.
    Hope its all sorted.

  • So the problem was tawk? If so, glad it's finally all sorted :-) I guess that means it's back to business as usual - nothing to see here.

  • DPDP Administrator, The Domain Guy
    edited January 2020

    /thread

    Edit: and kudos to the Doc.

This discussion has been closed.