Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


PSA: Telia L.A outage
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

PSA: Telia L.A outage

SplitIceSplitIce Member, Host Rep
edited August 2017 in Outages

Just a quick public service announcement for those Sys Admins (or technically adept users) scratching their heads.

We have been seeing issues with Telia in L.A since 2:42 AM (39 minutes ago). Confirmed with three L.A Telia sessions / networks. This isn't effecting us (beyond re-routing traffic via other upstreams and PoPs), but it's affecting many other providers.

I see one session behaving out of the 4 I have tested, and that's Choopa L.A.

Thanked by 1Fries

Comments

  • ClouviderClouvider Member, Patron Provider

    What problems exactly are being reported?

  • SplitIceSplitIce Member, Host Rep

    @Clouvider Peer failure. This is most obvious with networks that are single-homed Telia (such as Voxility L.A for protected ranges) where there is currently no routes.

  • J1021J1021 Member
    edited August 2017
    Thu, 3 Aug 2017 20:21:30 :
    We have a tech on site and confirmed that the temperature is normal. 
    The affected router has been rebooted but it is still down. 
    Further troubleshooting is in progress. 
    
    Thu, 3 Aug 2017 20:02:50 :
    We are suspecting a temperature issue.
    We do not have ETR at the moment. 
    
    Thu, 3 Aug 2017 19:29:12 :
    Our router in Los Angeles, USA is down, and the circuits connected to the downlink devices are also affected.
    Investigation is ongoing. 
  • qpsqps Member, Host Rep

    Voxility had a router failure at the same time.

  • SplitIceSplitIce Member, Host Rep
    edited August 2017

    @qps what @kcaj is echoing isn't Voxility it's Telia. Most people with Voxility protection are Singlehomed Telia meaning effectively they lost all their routes (this is why you multihome people).

    Thu, 3 Aug 2017 18:02:50 :
    We are suspecting a temperature issue.
    We do not have ETR at the moment.
    
    Thu, 3 Aug 2017 17:29:12 :
    Our router in Los Angeles, USA is down, and the circuits connected to the downlink devices are also affected.
    Investigation is ongoing.
    
    Best Regards,
    
    Customer Support Center
    Phone +46 771 19 11 70
    [email protected]
    
    www.teliacarrier.com
    
    Stockholm, Sweden
    Local free phone numbers:
    France 0800 917 453 | Germany 0800 182 7111 | Hong Kong 800 965 559 | Italy 800 789 798 | Netherlands 0800 022 9500
    Poland 00800 461 22 68 | Singapore 800 461 10 55 | Spain 900 984 631 | UK 0800 028 7406 | US 888 436 1133 | Canada 844 8730 909
    

    It's pretty major since they are still down.

  • qpsqps Member, Host Rep

    The Voxility NOC has told me that the issue is resolved, but it clearly is not since we are still seeing 60 to 90% packet loss.

  • SplitIceSplitIce Member, Host Rep
    edited August 2017

    @qps Voxility has resolved it via announcing other routes on routes that were previously restricted to Telia (I'm seeing heavy Comcast). Since Telia is most of their transit at DDoS levels there is likely some saturation.

  • qpsqps Member, Host Rep

    SplitIce said: Voxility has resolved it via announcing other routes on routes that were previously restricted to Telia (I'm seeing heavy Comcast). Since Telia is most of their transit at DDoS levels there is likely some saturation.

    There is heavy packet loss on their internal network between POPs. It could be a capacity issue.

  • @SplitIce said:
    @qps Voxility has resolved it via announcing other routes on routes that were previously restricted to Telia (I'm seeing heavy Comcast). Since Telia is most of their transit at DDoS levels there is likely some saturation.

    Well you can't blame them since they probably have smaller pipes coming from Comcast but I feel like they probably have other providers they can split the load with but they're not using it, because it can't be marked solved if you get 80% packet lost and high ping.

  • SplitIceSplitIce Member, Host Rep

    @Edmond they have NTT in some locations (unsure if L.A) but I doubt they would take any attacks there. Too pricey.

  • @SplitIce said:
    @Edmond they have NTT in some locations (unsure if L.A) but I doubt they would take any attacks there. Too pricey.

    Ya I figured other providers would get upset, but with all the traffic coming into Comcast, it's like a DDoS attack already to Comcast, aside from it's from legit users. I mean it's better than being down, but still you can hardy use it.

  • SplitIceSplitIce Member, Host Rep

    @Edmond Realistically it's the same everywhere when a big network in a popular region goes down. Right now Tinet is seeing alot more traffic from us, and could be exposed to bigger attacks (although I wouldn't expect issues since we have sane protection limits). If 300G normally goes over Telia in L.A thats 300G distributed elsewhere right now.

    It would be interesting to know how other comcast services in L.A are fairing. Hopefully fine. I'd expect Voxility are handing out nulls like free lollipops too right now anyway (due to reduced capacity).

    Thanked by 1jh_aurologic
  • @SplitIce said:
    @Edmond Realistically it's the same everywhere when a big network in a popular region goes down. Right now Tinet is seeing alot more traffic from us, and could be exposed to bigger attacks (although I wouldn't expect issues since we have sane protection limits). If 300G normally goes over Telia in L.A thats 300G distributed elsewhere right now.

    It would be interesting to know how other comcast services in L.A are fairing. Hopefully fine. I'd expect Voxility are handing out nulls like free lollipops too right now anyway (due to reduced capacity).

    That 300G goes somewhere else right? Well providers aren't going to buy insanely large bandwidth pipes that are left empty because it cost money, so it's really overloaded now. Wonder what makes Comcast to take the huge DDoS traffic into their networks anyway as it's causing a big load on it.

    These are one of the times where people should spend more money and get better protection from some other company that has backup peers like yours, can they even withstand 500G of DDoS anyway, it seems to be a unrealistic number.

  • SplitIceSplitIce Member, Host Rep

    @Edmond few single network regions can handle 500G unexpected anyway. Real 500G normally has to be distributed either by PoP or transit. No one (transit) just keeps that around for no reason.

Sign In or Register to comment.