Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Cloudflare route leak
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Cloudflare route leak

uptimeuptime Member
edited June 2019 in Outages

in case anyone's wondering what's up with the apparent downtime for many sites (including LET) ... something something cloudflare something something route leak something something verizon (?)

some details/discussion here -> Route Leak Impacting Cloudflare | Hacker News

EDIT2:

something something ASN 396531

EDIT3:

FHR said: $clueless_company leaked their routing table to retarded Verizon who leaked stuff to the rest of the internet.
TL;DR; half of the internet tried to reach each other through that $clueless_company, whose small ports couldn't handle the traffic => stuff died.

Thanked by 2dahartigan Janevski
Should I Use Teh Cloudflares To Make All My Web Stuff Go Fasterer ?
  1. Cloudflare?55 votes
    1. Yes
      27.27%
    2. No
      20.00%
    3. Maybe
        3.64%
    4. Anyone got any downers?
        1.82%
    5. REEEEEEEEEEEEEEEEEEEEEEEEE!!!!!
      10.91%
    6. BGP SELEKTER OK?
      14.55%
    7. debian, thx
      21.82%

Comments

  • ITLabsITLabs Member
    edited June 2019

    Cloudsh*t leaking pipes.... Discord is also affected.

  • HostYDHostYD Member, Host Rep
    edited June 2019
    Thanked by 1NanoG6
  • BopieBopie Member

    This wasn't limited to just cloud flare ;)

  • donlidonli Member

    Today's lesson in BGP - or how 1 rogue site can destroy people's interwebnetworking.

  • ...or start routing all of their traffic through china

  • jsgjsg Member, Resident Benchmarker

    Sorry, I have to criticize the poll. There is no "$7" option, neither "debian" and not even a "NVMe only!".

    Thanked by 3uptime ITLabs vimalware
  • verjinverjin Member

    Apparently the culprit was that "Allegheny Technologies Incorporated" announced a better-reachable route for Cloudflare through Verizon. (links to AS)

    Source:
    https://twitter.com/i/web/status/1143117619106652160
    https://twitter.com/i/web/status/1143113715119579137

    Thanked by 1uptime
  • donlidonli Member

    @tomahawkeer said:
    ...or start routing all of their traffic through china

    For those that don't know what that refers to...

    https://www.zdnet.com/article/for-two-hours-a-large-chunk-of-european-mobile-traffic-was-rerouted-through-china/

  • donlidonli Member
    edited June 2019

    @jsg said:
    Sorry, I have to criticize the poll. There is no "$7" option, neither "debian" and not even a "NVMe only!".

    BGP Sekektor OK ?

    BGP Selektor NOT OK ?

  • FHRFHR Member, Host Rep

    @verjin said:
    Apparently the culprit was that "Allegheny Technologies Incorporated" announced a better-reachable route for Cloudflare through Verizon. (links to AS)

    Source:
    https://twitter.com/i/web/status/1143117619106652160
    https://twitter.com/i/web/status/1143113715119579137

    Can confirm, that is absolutely true. $clueless_company leaked their routing table to retarded Verizon who leaked stuff to the rest of the internet.

    TL;DR; half of the internet tried to reach each other through that $clueless_company, whose small ports couldn't handle the traffic => stuff died.

    Thanked by 2uptime NanoG6
  • uptimeuptime Member

    I didn't like the poll results so I deleted them (by accident, but whatever)

    Feel free to cast your meaningless votes yet again if you care to

  • SplitIceSplitIce Member, Host Rep

    Seriously, who gave $clueless_company an unfiltered port.... oh Verizon.

    Thanked by 2Kris vimalware
  • KrisKris Member

    Likely caused by a poorly configured Noction route optimizer.

    Why Verizon allowed a tiny company out of Pittsburgh to announce the internet (no filters) is beyond me.

  • ITLabsITLabs Member

    @Kris said:
    Why Verizon allowed a tiny company out of Pittsburgh to announce the internet (no filters) is beyond me.

    That's because la fin est proche

  • donlidonli Member

    @ITLabs said:

    @Kris said:
    Why Verizon allowed a tiny company out of Pittsburgh to announce the internet (no filters) is beyond me.

    That's because la fin est proche

    On est Verizon on s'en fout.

    Thanked by 2ITLabs uptime
  • ralphralph Member

    assange's pic was awesome :(

    Thanked by 1uptime
  • Belgrade, Serbia - (BEG) Re-routed
    Casablanca, Morocco - (CMN) Re-routed
    Kigali, Rwanda - (KGL) Re-routed
    Islamabad, Pakistan (ISB) Re-routed
    Lahore, Pakistan - (LHE) Re-routed
    Rio de Janeiro, Brazil - (GIG) Re-routed
    Riyadh, Saudi Arabia - (RUH) Re-routed
    Perth, WA, Australia - (PER) Re-routed
    
    Past Incidents
    Jun 24, 2019
    Route Leak Impacting Cloudflare
    Resolved - Traffic levels have returned to normal now that the route leak has been fixed.
    We are now marking this incident as resolved. 
    Jun 24, 13:02 UTC
    Monitoring - The network responsible for the route leak has now fixed the issue.
    We are seeing improvement and are continuing to monitor this before we consider this issue resolved. 
    Jun 24, 12:42 UTC
    Update - This leak is impacting many internet services including Cloudflare.
    We are continuing to work with the network provider that created this route leak to remove it. 
    Jun 24, 12:34 UTC
    Update - We are continuing to work on a fix for this issue. 
    Jun 24, 11:43 UTC
    Identified - We have identified a possible route leak impacting some
    Cloudflare IP ranges and are working with the network involved to resolve this. 
    Jun 24, 11:36 UTC
    Investigating - Cloudflare is observing network related issues. 
    Jun 24, 11:02 UTC
    
    Thanked by 1uptime
  • How Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline Today

    What happened?

    Today at 10:30UTC, the Internet had a small heart attack. A small company in Northern Pennsylvania became a preferred path of many Internet routes through Verizon (AS701), a major Internet transit provider. This was the equivalent of Waze routing an entire freeway down a neighborhood street — resulting in many websites on Cloudflare, and many other providers, to be unavailable from large parts of the Internet. This should never have happened because Verizon should never have forwarded those routes to the rest of the Internet. To understand why, read on.

    Thanked by 3uptime Daniel15 Sofia_K
  • Yet another episode of drunk sysadmin.

Sign In or Register to comment.