Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Serious security issue at CloudFlare - CHANGE ALL YOUR PASSWORDS NOW.
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Serious security issue at CloudFlare - CHANGE ALL YOUR PASSWORDS NOW.

joepie91joepie91 Member, Patron Provider
edited February 2017 in General

Full details here: https://bugs.chromium.org/p/project-zero/issues/detail?id=1139

In summary:

  • Memory-related vulnerability in CloudFlare's reverse proxy software caused data to get mixed up.
  • Sensitive data (passwords, cryptographic keys, PII, and so on) ended up in Google's scrapes, and likely those of everybody else.
  • Assume all of your passwords and PII to be compromised; there's no reliable way to tell what sites were using CloudFlare when, or whether they were affected.

Change your passwords everywhere immediately, and keep an eye on your finances. Don't wait for notifications from vendors.

Some quotes from the disclosure thread:

"I didn't realize how much of the internet was sitting behind a Cloudflare CDN until this incident."

"Cloudflare pointed out their bug bounty program, but I noticed it has a top-tier reward of a t-shirt. Needless to say, this did not convey to me that they take the program seriously."

"Cloudflare did finally send me a draft. It contains an excellent postmortem, but severely downplays the risk to customers. They've left it too late to negotiate on the content of the notification."

"The examples we're finding are so bad, I cancelled some weekend plans to go into the office on Sunday to help build some tools to cleanup."

"We fetched a few live samples, and we observed encryption keys, cookies, passwords, chunks of POST data and even HTTPS requests for other major cloudflare-hosted sites from other users"

This is probably a good moment to refer back to the article I wrote last year, about how CloudFlare is actively putting the web at risk.

Seriously, stop using CloudFlare already. The only surprising thing about this incident is that it was accidental disclosure, instead of an active breach. This is playing with fire.

«1345

Comments

  • Hard to meme and make a joke about a serious issue despite the fact that 90% of LET would be using CF just for IPv4 network accessibility.

    Thanked by 1DewlanceVPS
  • In the last few days, I notice some abnormal notification in my accounts. Paypal said my account was logged in from another country; all of my Gmails need to sign in again. And now this, OMG.

  • jarjar Patron Provider, Top Host, Veteran
    Really impressed with Cloudflare's quick response, and how dedicated they are to cleaning up from this unfortunate issue. I told them I hoped they were able to get some sleep during the incident, and an engineer responded:
    
    "Its going to be a long weekend for us, but this is really important to us.  We need to protect as many people as we can and we need to make sure this can never happen again on our watch.".

    Great to see. I never expect anyone to be perfect or to not fail, my expectation is that they rush to the call and start working on resolution immediately.

    Thanked by 1netomx
  • NekkiNekki Veteran
    edited February 2017

    Do I have to do every password? I'm really busy today.

  • @jarland said:

    Really impressed with Cloudflare's quick response, and how dedicated they are to cleaning up from this unfortunate issue. I told them I hoped they were able to get some sleep during the incident, and an engineer responded:
    > 
    > "Its going to be a long weekend for us, but this is really important to us.  We need to protect as many people as we can and we need to make sure this can never happen again on our watch.".

    Great to see. I never expect anyone to be perfect or to not fail, my expectation is that they rush to the call and start working on resolution immediately.

    It does however raise second thoughts at running half the internet through a massive MITM reverse proxy.

    Thanked by 1WSS
  • @Nekki said:
    Do I have to do every password? I'm really busy okay.

    At the very minimum you should change passwords to the sites you don't need your wife finding out about.

  • @kcaj said:

    @Nekki said:
    Do I have to do every password? I'm really busy okay.

    At the very minimum you should change passwords to the sites you don't need your wife finding out about.

    That's the sort of out-of-the-box thinking I was hoping for.

  • jarjar Patron Provider, Top Host, Veteran

    kcaj said: It does however raise second thoughts at running half the internet through a massive MITM reverse proxy.

    It's not an invalid suggestion, but there are many schools of thought. I think the idea that you're never going to make a mistake yourself, or that you're going to partner with someone who is never going to make a mistake, is a bit naive.

    It's bad, no doubt. I'm glad someone with positive intent captured it before it spread through more malicious circles. We don't know that it didn't, but it clearly never reached critical mass if it did. Such a thing, if widely exploited, tends to make it's way to the surface eventually. Even if years later.

  • joepie91joepie91 Member, Patron Provider
    edited February 2017

    jarland said: I think the idea that you're never going to make a mistake yourself, or that you're going to partner with someone who is never going to make a mistake, is a bit naive.

    Right, but that's precisely why it's irresponsible to build such a massive single point of failure. Sooner or later, it's going to fail - and it did.

    @jarland said:

    Really impressed with Cloudflare's quick response, and how dedicated they are to cleaning up from this unfortunate issue. I told them I hoped they were able to get some sleep during the incident, and an engineer responded:
    > 
    > "Its going to be a long weekend for us, but this is really important to us.  We need to protect as many people as we can and we need to make sure this can never happen again on our watch.".

    Great to see. I never expect anyone to be perfect or to not fail, my expectation is that they rush to the call and start working on resolution immediately.

    Unfortunately, things get less rosy towards the end of the thread...

    I asked for a draft of their announcement, but they seemed evasive about it and clearly didn't want to do that. I'm really hoping they're not planning to downplay this. If the date keeps extending, they'll reach our "7-day" policy for actively exploited attacks.

    They gave several excuses that didn't make sense, then asked to speak to me on the phone to explain. They assured me it was on the way and they just needed my PGP key. I provided it to them, then heard no further response.

    Cloudflare did finally send me a draft. It contains an excellent postmortem, but severely downplays the risk to customers.

  • jarjar Patron Provider, Top Host, Veteran

    This is my favorite part so far:

    image

  • I have 2FA enabled on my PayPal, iCloud and Google accounts, and other companies such as my phone and cable company don't use cloudflare so hopefully for people who have taken measures like me, we aren't affected.

  • massive single point of failure

    Was just thinking that. Until the next best thing comes along...

  • mfsmfs Banned, Member
    edited February 2017

    I've checked various accounts of mine and only LET currently uses cloudflare on the login page..

    ethancedrik said: PayPal, iCloud and Google

    none of those use cloudflare afaik

  • mfs said: none of those use cloudflare afaik

    at one point in time and still under some circumstances I've used the same password on multiple accounts so that's why I was worried about locking those down, but really the amount of websites I use that use cloudflare are minimal

  • Cancer.

  • mfsmfs Banned, Member
    edited February 2017

    Uber, 1Password, FitBit, OKCupid,

    I don't use any of those; the 1password thing reminds me of a different thread we had here, where people presented their password managers of choice

    ethancedrik said: same password on multiple accounts

    I don't know any my passwords, keepassx0 ftw

    To be honest I've checked 1password.com and they do not seem to use cloudflare, at least now

    Thanked by 1vimalware
  • ethancedrik said: at one point in time and still under some circumstances I've used the same password on multiple accounts

    I sometime use the same password on a few sites too , it's only site that if my account/password is compromised not a major issue as the account got very limited/no personal info and not a lot of damage can be done.

    But for any site with personal info or financial info i use a unqie password.

  • I suppose anyone 'hiding' their IP behind CF should now consider it compromised too? Not sure of the ins and outs, but it seems that way.

  • Ok. I think making a list of providers whose WHMCS and Solus/other panels use Cloudflare would be a good idea , to start the password reset marathon. (do it in one 30 minute session maybe)

    Perhaps a separate (sticky?) thread with a link to a Google sheet after we have a nice list?

    I'm not worried about password reuse attacks since I use unique randomly generated passwords, but it would save people time, to compile a list of who's on Cloudflare. (or may have been in the past during account signup , providers feel free to post this info.)

    I'll start : (scanning domains from cpanel urls from my keepass providers folder)

    Just vps providers for now.

    • Hosthatch: portal.hosthatch.com
    • ImpactVPS/subnetlabs : subnetlabs.com (billing and custom provisioning panel )
    • Wishosting wishosting.com (only this whmcs domain is affected)
    • cociu/hostsolutions panel : secure.hostsolutions.ro

    I won't edit this post. I'll just post a new comment with 3+ new unique affected providers .

  • wow good thing i use the same password and no 2 step verification on all my accounts

    /s

  • Ramnode's clientarea (Not SolusVM) seems to use cloudflare?
    (confirm if not true)

    shit, its pretty much the only thing i can think ive used.

    so, is it safe to change the pass now, or should something else be done/wait? (its fuckin 4pm, too tired to read the report stuff rn)

  • MikeAMikeA Member, Patron Provider
    edited February 2017

    I read CloudFlare's article regarding it and was thinking, "well, this doesn't sound that bad", then I read the Travis guys report and was like "well, this is fucking bad!"

    Good thing I don't use CloudFlare I guess. Props to Travis @ Google for putting this out there and pushing CF, it's definitely pretty bad.

  • I hate to say it, but these were incredibly trivial coding errors. I don't care what the hell you claim is automatically generated, and what isn't. These folks obviously aren't used to allocating and deallocating memory by themselves, and this frightens the hell out of me by just how much they've built a business sticking themselves between the public internet and business.

    Waiting for the first Node exploit that takes down Web 2.1 so we can go back to the good old days. Namesilo and I will be just fine.

  • joepie91joepie91 Member, Patron Provider

    @ricardo said:
    I suppose anyone 'hiding' their IP behind CF should now consider it compromised too? Not sure of the ins and outs, but it seems that way.

    Trying to use CF to "hide an IP" is a terrible idea in the first place, but I doubt that this bug would have disclosed any origins (since it seemed to just mix up the content). It's a memory-related vulnerability however, so there's no way to be 100% sure.

    Pandy said: Ramnode's clientarea (Not SolusVM) seems to use cloudflare? (confirm if not true)

    Correct.

  • mfsmfs Banned, Member
    edited February 2017

    Pandy said: Ramnode's clientarea (Not SolusVM) seems to use cloudflare? (confirm if not true)

    yep, clientarea.ramnode.com uses cloudflare's MiTM

    Pandy said: so, is it safe to change the pass now

    Honestly I'd expect my service provider to turn off cloudflare on the login page; anyway the issue has been solved on "2017-02-21 1803 UTC" according to the cloudflare's blog page.

    Thanked by 2Pandy WSS
  • dig NS +short ramnode.com
    isla.ns.cloudflare.com.
    tom.ns.cloudflare.com.

    dig NS +short lowendtalk.com
    ken.ns.cloudflare.com.
    ruth.ns.cloudflare.com.

    dig em all...

  • @ethancedrik said:
    I have 2FA enabled on my PayPal, iCloud and Google accounts, and other companies such as my phone and cable company don't use cloudflare so hopefully for people who have taken measures like me, we aren't affected.

    Have the same done on my accounts - I love the iCloud/Apple 2fa system :)

  • AbdussamadAbdussamad Member
    edited February 2017

    ricardo said:

    dig NS +short ramnode.com isla.ns.cloudflare.com. tom.ns.cloudflare.com.

    dig NS +short lowendtalk.com ken.ns.cloudflare.com. ruth.ns.cloudflare.com.

    dig em all..

    you need to look at the A records because you can turn proxying on and off selectively for different sub-domains. if the A record returns a cloudflare ip address then proxying is on. you can tell its a cloudflare IP by doing a whois <ipaddress> under linux.

    inception hosting's client area is also affected BTW.

Sign In or Register to comment.