Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Anycast routes latency.
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Anycast routes latency.

umiumi Member
edited July 2020 in Help

Why sometimes anycast paths have terrible delays to certain providers/datacenters?
Is this has to do something with a lack of peering?
On traceroute, it seems like packets literally running circles around.
151 ms to L.A., 122ms to Dallas, 94ms to VA...

Canada, BC, Vancouver   Telus           0%  33  5.59    6.06    5.46    6.68    0.51    
Canada, BC, Vancouver   Shaw            0%  33  16.45   15.31   13.12   18.36   1.45    
USA, CA, Fremont    Hurricane FMT2   0%     33  9.65    9.59    9.53    9.72    0.04
USA, CA, Fremont    Linode            0%    33  25.66   25.82   25.24   26.76   0.4 
USA, CA, San Francisco Digital Ocean    0%  33  22.27   22.89   22.22   23.33   0.41 
USA, CA, Los Angeles    QuadraNET    0%     33  151.5   151.65  151.11  152.59  0.33 
USA, CA, Los Angeles    Vultr           0%  34  1.15    1.26    1.1     4.79    0.62 
USA, CA, Seattle    Google            0%    33  25  25.03   24.43   25.47   0.21    
USA, CO, Denver     Cogent           0%     32  23.77   23.68   22.78   24.23   0.43    
USA, TX, Dallas     Softlayer           0%  33  122.2   123.39  121.83  129.3   2.13
USA, IL, Chicago    Cogent           0%     32  13.89   15.83   13.79   25.04   2.59    
USA, GA, Atlanta    55 Marietta/RamNode0%   33  15.24   15.04   14.61   16.06   0.25    
USA, VA, Vint Hill  OVH             6.3%    32  94.09   92.61   90.73   94.9    1.47        
USA, NY, New York Telehouse/RamNode  0%     32  17.24   17.36   17.09   18.17   0.31 
Canada, QC, Montreal    OVH             0%  32  96.04   94.73   93.57   96.15   0.94

This one is especially terrible with across the pond jumps.
(an .icu TLD nameserver actually)

Canada, BC, Vancouver   Telus   0%  15  143.27  143.63  143.22  144.32  0.32    show    
Canada, BC, Vancouver   Shaw    0%  14  158.71  157.56  155.39  159.95  1.75    show    
USA, CA, Fremont    Hurricane FMT2  0%  15  59.57   59.79   59.41   60.65   0.44    show    
USA, CA, Fremont    Linode  0%  15  192.55  192.95  192.52  193.58  0.38    show    
USA, CA, San Francisco  Digital Ocean   0%  15  193.94  193.59  193.11  194.11  0.32    show    
USA, CA, Los Angeles    QuadraNET   0%  15  57.38   57.47   57.1    58.07   0.27    show    
USA, CA, Los Angeles    Vultr   0%  15  187.48  187.28  186.63  188.58  0.44    show    
USA, CA, Seattle    Google  0%  15  57.49   57.48   56.95   57.84   0.25    show    
USA, CO, Denver     Cogent  0%  14  26.91   26.31   24.84   26.96   0.82    show    
USA, TX, Dallas     Softlayer   0%  15  121.22  121.79  121.14  123.44  0.58    show    
USA, IL, Chicago    Cogent  0%  14  2.86    3.54    1.99    7.61    1.69    show    
USA, GA, Atlanta    55 Marietta/RamNode     0%  15  18.07   18.63   18.07   19.14   0.39    show    
USA, VA, Vint Hill  OVH     0%  14  208.36  208.01  207.39  208.62  0.34    show    
USA, NY, New York   Telehouse/RamNode   0%  14  2.17    2.15    1.7     2.2     0.13    show    
Canada, QC, Montreal    OVH     0%  14  203.62  204.16  203.62  205.32  0.4     show

Comments

  • sanvitsanvit Member

    iirc Anycast will route you through the shortest route (least hop) unless manually overriden, and shortest route doesn't always mean smallest ping

    Thanked by 2vimalware dedicados
  • ClouviderClouvider Member, Patron Provider
    edited July 2020

    @sanvit said:
    iirc Anycast will route you through the shortest route (least hop) unless manually overriden, and shortest route doesn't always mean smallest ping

    Shortest ASN path. But it boils down to the same thing. Really hard to manage if you have inconsistent set of upstreams

  • umiumi Member
    edited July 2020

    BuyVM's homemade anycast looks quite nice overall:
    All within 45ms in US and EU, except Shaw ;)
    Francisco, please please add Singapore!

    Canada, BC, Vancouver   Telus   0%  48  41.05   41.87   41.05   42.47   0.45    show    
    Canada, BC, Vancouver   Shaw    2.1%    48  80.54   82.72   77.71   178.81  14.28   show    
    USA, CA, Fremont    Hurricane FMT2  0%  48  20.2    20.28   20.14   22.38   0.34    show    
    USA, CA, Fremont    Linode  0%  48  22.2    22.01   21.25   23.82   0.61    show    
    USA, CA, San Francisco  Digital Ocean   0%  48  21.09   21.08   20.16   21.3    0.18    show    
    USA, CA, Los Angeles    QuadraNET   0%  48  8.23    8.41    7.87    11.47   1   show    
    USA, CA, Los Angeles    Vultr   0%  48  8.84    8.84    8.1     9.95    0.23    show    
    USA, CA, Seattle    Google  0%  48  41.05   40.92   39.89   55.36   2.16    show    
    USA, CO, Denver     Cogent  0%  47  42.03   42.58   41.74   44.84   0.91    show    
    USA, TX, Dallas     Softlayer   0%  48  42.17   43  42.06   48.23   1.07    show    
    USA, IL, Chicago    Cogent  0%  47  22.88   23.17   20.56   30.46   2.42    show    
    USA, GA, Atlanta    55 Marietta/RamNode     0%  48  19.09   18.69   17.94   27.08   1.43    show    
    USA, VA, Vint Hill  OVH     0%  47  8.57    9.02    8.2     17.21   1.35    show    
    USA, NY, New York   Telehouse/RamNode   0%  47  14.07   2.58    2.12    14.07   1.87    show    
    Canada, QC, Montreal    OVH     0%  46  9.82    9.9     8.77    13.14   0.64    show    
    France, Paris   Online.net  0%  45  10.03   10.18   9.96    14.45   0.7     show    
    
    Thanked by 1stefeman
  • umiumi Member
    edited July 2020

    @Clouvider said:

    @sanvit said:
    iirc Anycast will route you through the shortest route (least hop) unless manually overriden, and shortest route doesn't always mean smallest ping

    Shortest ASN path. But it boils down to the same thing. Really hard to manage if you have inconsistent set of upstreams

    If the staff too tired with manual overrides we see jumps across the ocean.
    Wonder if Cloudflare has a whole department fighting anycast troubles constantly...

    Thanked by 1vimalware
  • trewqtrewq Administrator, Patron Provider

    @umi said:

    @Clouvider said:

    @sanvit said:
    iirc Anycast will route you through the shortest route (least hop) unless manually overriden, and shortest route doesn't always mean smallest ping

    Shortest ASN path. But it boils down to the same thing. Really hard to manage if you have inconsistent set of upstreams

    If the staff too tired with manual overrides we see jumps across the ocean.
    Wonder if Cloudflare has a whole department fighting anycast troubles constantly...

    You don't have to do nearly as many manual tweaks when you're on almost every major IX in the world.

    Thanked by 1vimalware
  • FranciscoFrancisco Top Host, Host Rep, Veteran

    @umi said:
    BuyVM's homemade anycast looks quite nice overall:
    All within 45ms in US and EU, except Shaw ;)
    Francisco, please please add Singapore!

    > Canada, BC, Vancouver     Telus   0%  48  41.05   41.87   41.05   42.47   0.45    show    
    > Canada, BC, Vancouver     Shaw    2.1%    48  80.54   82.72   77.71   178.81  14.28   show    
    > USA, CA, Fremont  Hurricane FMT2  0%  48  20.2    20.28   20.14   22.38   0.34    show    
    > USA, CA, Fremont  Linode  0%  48  22.2    22.01   21.25   23.82   0.61    show    
    > USA, CA, San Francisco    Digital Ocean   0%  48  21.09   21.08   20.16   21.3    0.18    show    
    > USA, CA, Los Angeles  QuadraNET   0%  48  8.23    8.41    7.87    11.47   1   show    
    > USA, CA, Los Angeles  Vultr   0%  48  8.84    8.84    8.1     9.95    0.23    show    
    > USA, CA, Seattle  Google  0%  48  41.05   40.92   39.89   55.36   2.16    show    
    > USA, CO, Denver   Cogent  0%  47  42.03   42.58   41.74   44.84   0.91    show    
    > USA, TX, Dallas   Softlayer   0%  48  42.17   43  42.06   48.23   1.07    show    
    > USA, IL, Chicago  Cogent  0%  47  22.88   23.17   20.56   30.46   2.42    show    
    > USA, GA, Atlanta  55 Marietta/RamNode     0%  48  19.09   18.69   17.94   27.08   1.43    show    
    > USA, VA, Vint Hill    OVH     0%  47  8.57    9.02    8.2     17.21   1.35    show    
    > USA, NY, New York     Telehouse/RamNode   0%  47  14.07   2.58    2.12    14.07   1.87    show    
    > Canada, QC, Montreal  OVH     0%  46  9.82    9.9     8.77    13.14   0.64    show    
    > France, Paris     Online.net  0%  45  10.03   10.18   9.96    14.45   0.7     show    
    > 

    Not a fan of their free speech laws.

    Tokyo?

    Francisco

  • umiumi Member
    edited July 2020

    Tokyo is just 120ms away from LasVegas. If Singapore is not an option then maybe Malaysia or Sydney.
    Malaysia has Thailand,Vietnam,Indonesia,India and even Perth within 50-60ms reach. The population is crazy huge out there.
    Sydney has all Australia covered plus New Zealand, Fiji and a whole bunch of Pacific islands with exception of French Polynesia which is connected via Hawaii.

    Singapore   Digital Ocean   0%  8   181.54  181.45  180.88  181.92  0.34    show    
    Japan, Tokyo    Vultr   0%  8   122.13  124.24  121.89  137.4   4.99    show    
    Australia, Sydney   Vultr   0%  8   157.35  157.29  156.65  158.6   0.57    show    
    Taiwan, Taichung    Google  0%  8   149.65  150.14  149.65  150.61  0.28    show    
    
  • FranciscoFrancisco Top Host, Host Rep, Veteran

    @umi said:
    Tokyo is just 120ms away from LasVegas. If Singapore is not an option then maybe Malaysia or Sydney.
    Malaysia has Thailand,Vietnam,Indonesia,India and even Perth within 50-60ms reach. The population is crazy huge out there.
    Sydney has all Australia covered plus New Zealand, Fiji and a whole bunch of Pacific islands with exception of French Polynesia which is connected via Hawaii.

    > Singapore     Digital Ocean   0%  8   181.54  181.45  180.88  181.92  0.34    show    
    > Japan, Tokyo  Vultr   0%  8   122.13  124.24  121.89  137.4   4.99    show    
    > Australia, Sydney     Vultr   0%  8   157.35  157.29  156.65  158.6   0.57    show    
    > Taiwan, Taichung  Google  0%  8   149.65  150.14  149.65  150.61  0.28    show    
    > 

    Fairly sure Malaysia is in a similar spot for speech.

    AU is interesting, would have to test the latency to see if it would be good.

    I got a good offer in Singapore but I think it would be a bit hypocritical to go on about freedom of speech then host in a country without it.

    Francisco

  • SplitIceSplitIce Member, Host Rep
    edited July 2020

    Optimizing Anycast routes is challenging and time consuming. Not everyone does a good job. Not every route is 'fixable' although 90%+ are. It gets even more difficult when the other networks are heavily (manually) tuned or Anycast themselves.

    @umi if you are a customer have you thought to point out bad routes to the provider of the network? Sometimes people are just unaware of a route / configuration change (particularly in the low end space).

    Thanked by 1vimalware
  • SplitIceSplitIce Member, Host Rep

    @trewq said: You don't have to do nearly as many manual tweaks when you're on almost every major IX in the world.

    I'm betting they don't have to deal with as much bullshit from ISPs that refuse to peer as well.

  • zenkizenki Member

    @Francisco said:
    AU is interesting, would have to test the latency to see if it would be good.

    Yes, please consider AU :)

  • tetechtetech Member

    @HyperK9 said:

    @Francisco said:
    AU is interesting, would have to test the latency to see if it would be good.

    Yes, please consider AU :)

    +1

  • umiumi Member

    A server in Singapore can serve> @SplitIce said:

    Optimizing Anycast routes is challenging and time consuming. Not everyone does a good job. Not every route is 'fixable' although 90%+ are. It gets even more difficult when the other networks are heavily (manually) tuned or Anycast themselves.

    @umi if you are a customer have you thought to point out bad routes to the provider of the network? Sometimes people are just unaware of a route / configuration change (particularly in the low end space).

    It looks like a terrible issue with anycast providers and a business model for some.
    They are well aware of it, and continue on selling xx anycast PoPs...
    It's easier to name a few providers that have done ipv4 anycast right:

    I use some .club and .co domains. Their TLD namservers' anycast ips are from ULTRADNS - NeuStar, Inc. ASN and have no big routing issues worldwide. I can recommend them.

    Cloudflare anycast nameservers that I managed to test are fast and have no big routing issues except once a year July downtime ;) Best of free dns options.

    I'm testing BuyVM's anycast and I'm gonna use it in production soon - it is good and uniform, but the South-EastAsia/Australia regions have a blind spot with 160-180ms RTT to nearest server. For just a 3 of them it is a masterpiece I think.

    The geek-solutions.ca came here asking for test of the new anycast product, I signed in, but still nothing heard from them for many weeks.

    The .icu .xyz anycast nameservers are operated by CENTRALNIC-ANYCAST-E, CENTRALNIC-ANYCAST-F ASNs and are very bad at routing.

    CLOUDNSNET has some issues in US on their Premium S tier anycast nameservers.

    VULTR anycast nameservers have some routing issues worldwide, but overall, it is a decent free dns service.

    HE.net can do AXFR but the only decent anycast nameserver is ns4.he.net. Others have routing issues in NewYork and other places. Not actually that bad for a free service. ns4.he.net can be mixed with your own nameservers.

  • SplitIceSplitIce Member, Host Rep

    @umi you can always use our Anycast.

    Thanked by 1umi
  • klikliklikli Member

    @Francisco said: Tokyo?

    Please do it, sir :)

    I got the same reasoning as you - SG's freedom of speech stance is really concerning.

    Tokyo on the other hand is much better - very few competitions there (Vultr/Linode), decent latency to US. My head just aches when I see traceroutes from SG doing SG -> US -> EU resulting in 300ms+ latency.

    And your favourite transit providers (Cogent + HE) are both present in JP although neither of them has a strong Asian presence.

  • FranciscoFrancisco Top Host, Host Rep, Veteran

    @klikli said: And your favourite transit providers (Cogent + HE) are both present in JP although neither of them has a strong Asian presence.

    Thanks boss, I appreciate it.

    Honestly I'd probably prefer to buy into someone elses network for an Asia roll out.

    That, or just whore myself to NTT.

    Francisco

  • klikliklikli Member
    edited July 2020

    @Francisco said: Honestly I'd probably prefer to buy into someone elses network for an Asia roll out.

    That, or just whore myself to NTT.

    Yeah, that's easier probably easier, except you will now have tickets after tickets begging you to fix their anycast routing since the bandwidth mix are different among the 4 locations :(

  • FranciscoFrancisco Top Host, Host Rep, Veteran

    @klikli said: Yeah, that's easier probably easier, except you will now have tickets after tickets begging you to fix their anycast routing since the bandwidth mix are different among the 4 locations

    Very true!

    Who knows what will happen, but we need to upgrade LU to Ryzen before we move into new digs :)

    Francisco

  • umiumi Member

    If the anycast routing stability will be better in Tokyo then sure go for it!

  • FranciscoFrancisco Top Host, Host Rep, Veteran

    @umi said:
    If the anycast routing stability will be better in Tokyo then sure go for it!

    For what its worth our anycast will be handled by Cloudflare sometime this year. They intend for it to be purely latency driven, basically argo optimized anycast ips.

    Neat right?

    Francisco

  • @Francisco said:

    @umi said:
    If the anycast routing stability will be better in Tokyo then sure go for it!

    For what its worth our anycast will be handled by Cloudflare sometime this year. They intend for it to be purely latency driven, basically argo optimized anycast ips.

    Neat right?

    Francisco

    Asia location would sure be nice. :P Some alternative for OVH Anti-DDoS.

  • SplitIceSplitIce Member, Host Rep

    @stefeman we are in Asia (Singpaore and Tokyo) and are regularly used expecially in Singapore a protection for peoples OVH servers :)

  • @SplitIce said:
    @stefeman we are in Asia (Singpaore and Tokyo) and are regularly used expecially in Singapore a protection for peoples OVH servers :)

    Do you offer AnyCast without DDoS mitigation? The bandwidth requirement for me with AnyCast is outside the budget of what I would need but I have looked X4B many times on and off.

  • SplitIceSplitIce Member, Host Rep

    @daxterfellowes

    No we don't. We used to do non-protected services 5+ years ago... and they constantly got attacked. I sleep alot better now.

  • @umi said:
    A server in Singapore can serve> @SplitIce said:

    Optimizing Anycast routes is challenging and time consuming. Not everyone does a good job. Not every route is 'fixable' although 90%+ are. It gets even more difficult when the other networks are heavily (manually) tuned or Anycast themselves.

    CLOUDNSNET has some issues in US on their Premium S tier anycast nameservers.

    Could you expand further on your experiences with CloudDNSnet anycast/GeoDNS by chance?

  • umiumi Member
    edited July 2020

    Responce time >90ms within US for a nameserver is a bit slow. But with some not so small providers it is so:
    Here is the check of their nameservers (from control panel) with ping.pe:
    all 4 nameservers show same slowdown in L.A. , Dallas, VintHill

    Canada, BC, Vancouver   Telus   0%  12  6.62    6.17    5.53    6.65    0.48    show    
    Canada, BC, Vancouver   Shaw    0%  12  14.21   15.1    13.19   16.38   1.01    show    
    USA, CA, Fremont    Hurricane FMT2  0%  12  9.57    9.57    9.54    9.63    0.03    show    
    USA, CA, Fremont    Linode  0%  12  26.02   25.8    25.22   26.47   0.35    show    
    USA, CA, San Francisco  Digital Ocean   0%  12  23.2    22.96   22.24   23.25   0.34    show    
    USA, CA, Los Angeles    QuadraNET   0%  13  151.08  151.5   151.08  152.16  0.38    show    
    USA, CA, Los Angeles    Vultr   0%  13  1.13    1.17    1.13    1.24    0.03    show    
    USA, CA, Seattle    Google  0%  12  24.74   25.06   24.74   25.59   0.23    show    
    USA, CO, Denver     Cogent  0%  12  23.82   23.66   22.73   25.3    0.7     show    
    USA, TX, Dallas     Softlayer   0%  12  122.33  123     122.28  125.25  0.76    show    
    USA, IL, Chicago    Cogent  0%  12  16.8    17.77   13.59   35.85   5.82    show    
    USA, GA, Atlanta    55 Marietta/RamNode     0%  12  15.17   14.94   14.75   15.17   0.09    show    
    USA, VA, Vint Hill  OVH     0%  12  91.74   91.21   90.75   91.74   0.3     show    
    USA, NY, New York   Telehouse/RamNode   0%  12  17.12   17.41   17.12   18.07   0.28    show    
    Canada, QC, Montreal    OVH     0%  12  91.98   92.17   91.85   92.68   0.23    show
    

    Here is the Cloudflare's nameserver to compare:
    Just a little glitch with HE in Freemont.

     Canada, BC, Vancouver  Telus   0%  9   2.39    2.33    2.27    2.41    0.04    show    
    Canada, BC, Vancouver   Shaw    0%  9   13.09   14.55   13.08   17.96   1.69    show    
    USA, CA, Fremont    Hurricane FMT2  0%  9   69.17   69.04   68.95   69.19   0.08    show    
    USA, CA, Fremont    Linode  0%  9   1.14    1.16    1.14    1.21    0.02    show    
    USA, CA, San Francisco  Digital Ocean   0%  9   1.09    1.09    1.04    1.12    0.02    show    
    USA, CA, Los Angeles    QuadraNET   0%  9   12.41   12.06   11.47   12.48   0.43    show    
    USA, CA, Los Angeles    Vultr   0%  9   7.71    7.82    7.64    8.26    0.17    show    
    USA, CA, Seattle    Google  0%  9   7.63    7.61    7.55    7.72    0.05    show    
    USA, CO, Denver     Cogent  0%  9   3.77    2.91    2.26    3.77    0.37    show    
    USA, TX, Dallas     Softlayer   0%  9   2.29    2.29    2.27    2.35    0.02    show    
    USA, IL, Chicago    Cogent  0%  9   3.81    3.25    1.88    4.81    0.87    show    
    USA, GA, Atlanta    55 Marietta/RamNode     0%  9   1.49    1.15    1.1     1.49    0.12    show    
    USA, VA, Vint Hill  OVH     0%  9   2.19    2.22    2.18    2.34    0.05    show    
    USA, NY, New York   Telehouse/RamNode   0%  9   1.17    1.82    1.08    4.31    1.32    show    
    Canada, QC, Montreal    OVH     0%  9   8.76    8.77    8.75    8.83    0.02    show    
    
Sign In or Register to comment.