A serious question about Cogent Communications

natestamm

I could really use some word from the community.

Having Traceroutes fail over .cogentco Hops.

Only begins after a period of time. New provider and IP same thing. Good quality connection then bombs after time. Using MTR to analyze this from my end.

Any and all opinions welcome here, thanks!

rds100

Does the end to end connectivity break, or just the traceroutes stop working?

natestamm

@rds100 I am getting a large amount of packet loss on certain hops. In some cases it does become unreachable.

klikli

@natestamm said: I am getting a large amount of packet loss on certain hops.

EDIT: Sorry for the confusion. I thought you meant a specific hop is suffering from packet loss/down.
~~
Sorry for bad formatting, please head over http://cluepon.net/ras/traceroute.pdf for the original text.

Depending on the design of the router, the “Slow Path” of ICMP generation may be handled by a dedicated CPU (on larger routers these are often di stributed onto each line - card), or it may share the same CPU as the Control Plane. Either way, ICMP generation is considered to be one of the lowest priority functions of the router, and is typically rate - limited to very low values (typically in the 5 0 - 300 packets/sec range , depending on the device ). When ICMP generation occurs on the same CPU as the Control Plane, you may notice that events like BGP churn or even heavy CLI use (such as running a computationally intensive “show” command) will noticeably del ay the ICMP generation process. One of the biggest examples of this type of behavior is the Cisco “BGP Scanner” process, which runs every 60 seconds to perform internal BGP maintenance functions on most IOS based devices. On a router without a dedicated “D ata Plane - Slow Path” CPU (such as on the popular Cisco 6500 and 7600 - series platforms) , Traceroute users will notice latency spikes on the device every 60 seconds as this scanner runs. It is important to note that this type of latency spike is cosmetic, and only affects Traceroute (by delaying the returning ICMP TTL Exceed packet) . Traffic forwarded through the router via the normal “Fast Path” method will be unaffected. An easy way to detect these cosmetic spikes is to look at the future hops in the Trac eroute. If there was a real issue causing increased latency or packet loss on all traffic forwarded through the interface, then by definition you would see the cumulative effects in all future Traceroute hops. If you see a latency spike on one Traceroute h op, and it goes away in the next hop, you can easily deduce that the spike was actually caused by a cosmetic delay in the ICMP generation process.

natestamm

@klikli thanks for the response. I am actually trying to relate this and the mentioned monitoring in regards to an issue with primary IPs timing out for a particular host. Could this cause one of our main IPs to be unresponsive for a few days? I am not trying to be sarcastic I honestly have been running some intensive CLI processes lately But really nothing I would consider extraordinary. My point is I was asked to watch these Traceroutes as this particular host has not gotten back to me with reason why our primary IP is timing out constantly for myself and users all over the US. It effectively brought down the services we were using. And the two other addresses allocated to that particular machine are Extremely slow.


*Any ways a better update is just that a new machine I thought may have been suffering similar issues is working out good for me at the moment. Really I just get paranoid over possible denial of service when I see things like this. Meh nothing to stress over so far. Thanks again fellas.

raindog308

Someone once said to me that Cogent is the USPS Media Mail of the Internet world. If you want Fedex Next Day Morning Delivery service, go with Level3, etc. If you want best effort bulk rate, go with Cogent.

That said, I'm just repeating something I heard.

qps

Cogent can do some strange things, especially around peak usage time. For instance, they have peering with most carriers in Atlanta, but during peak hours, some of the Atlanta peering fills up, so they start routing excess traffic through Washington DC, Dallas, or Miami depending on the carrier. They are still a pretty good provider for the money. Cogent's NOC/support is one of the best I've dealt with.

superpilesos

Try using Cogent in Europe to each UPC in Europe. The results will not be good.

superpilesos

Tracing route to upc.cz [213.46.242.71]
over a maximum of 30 hops:

  3    19 ms    19 ms    20 ms  te0-2-0-1.ccr21.bts01.atlas.cogentco.com [130.11
7.3.69]
  4    27 ms    27 ms    26 ms  te0-2-0-2.ccr21.muc01.atlas.cogentco.com [154.54
.38.237]
  5    32 ms    32 ms    33 ms  te0-3-0-5.mpd21.fra03.atlas.cogentco.com [130.11
7.50.233]
  6    42 ms    41 ms    42 ms  te0-1-0-2.mpd21.par01.atlas.cogentco.com [130.11
7.0.102]
  7   117 ms   116 ms   116 ms  te0-6-0-2.mpd21.jfk02.atlas.cogentco.com [154.54
.46.109]
  8   116 ms   117 ms   117 ms  te0-1-0-1.ccr21.jfk05.atlas.cogentco.com [154.54
.31.2]
  9   109 ms   110 ms   110 ms  xe-9-0-0.edge1.NewYork1.Level3.net [4.68.111.45]

 10   112 ms   112 ms   112 ms  UPC-BROADBA.edge1.NewYork1.Level3.net [4.78.164.
222]
 11   113 ms   112 ms   113 ms  84.116.137.189
 12   118 ms   118 ms   119 ms  84.116.136.157
 13     *      119 ms   118 ms  nl-ams03a-mc1-gi-5-1.aorta.net [213.46.161.194]

 14   119 ms   119 ms   119 ms  213.46.242.71

Trace complete.

Microlinux

@qps said: Cogent's NOC/support is one of the best I've dealt with.

I'll second that. It's nice to get right to engineer who can actually do something and knows what you are talking about vs a screener who poorly translates your problem to a engineer who calls back an hour later.

dylziez

@qps said: Cogent's NOC/support is one of the best I've dealt with.

Their sales is one of the most aggressive, obnoxious I have ever dealt with. And you can't even tell them to never contact you again because turnover is so high it's not going to be the same person anyways.

qps

@dylziez said: Their sales is one of the most aggressive, obnoxious I have ever dealt with. And you can't even tell them to never contact you again because turnover is so high it's not going to be the same person anyways.

You can have them put you on their do not contact list. We've done that in the past. Not sure what the process is exactly, but the calls/e-mails seemed to stop after I asked for this back a few years ago.

concerto49

@qps said: Cogent can do some strange things, especially around peak usage time. For instance, they have peering with most carriers in Atlanta, but during peak hours, some of the Atlanta peering fills up, so they start routing excess traffic through Washington DC, Dallas, or Miami depending on the carrier.

This is when you need route optimization and use Cogent when it's good.

William

route optimizers do not detect latency changes, for example if comcast in ATL breaks and Cogent reroutes Comcast traffic over MIA the AS path is the same (just different fiber path/L2) and the route optimizer can't pick another one.

(There are ways around this like do a trace on any prefix in BGP but obviously this is not feasible due to massive amount of CPU power required to keep this list updated (imagine scanning 300k netblocks every 60 seconds))

concerto49

@William said: route optimizers do not detect latency changes, for example if comcast in ATL breaks and Cogent reroutes Comcast traffic over MIA the AS path is the same (just different fiber path/L2) and the route optimizer can't pick another one.

But isn't the whole point of something like Noction or Internap FCP to make the lowest latency, least packet loss and fastest route?

Serghei

@William said:
route optimizers do not detect latency changes, for example if comcast in ATL breaks and Cogent reroutes Comcast traffic over MIA the AS path is the same (just different fiber path/L2) and the route optimizer can't pick another one.

Route optimizers like Noction IRP are designed to influence the inter-domain routing rather than intra-domain. Therefore it will select the best-performing AS-path for the traffic to flow. It has no ability to influence the routing inside a Service Provider’s network.

(There are ways around this like do a trace on any prefix in BGP but obviously this is not feasible due to massive amount of CPU power required to keep this list updated (imagine scanning 300k netblocks every 60 seconds))

In order to efficiently use the router's CPU power, IRP choses the most relevant prefixes for probing. If IRP gets the traffic data by a NetFlow/sFlow feed, then it will sort the prefixes by relevancy criteria (like volume of traffic exchanged with your network). If IRP gets traffic data by port mirroring, then it will additionally analyze various network anomalies such as excessive delay and high packet retransmits in order to figure out the potentially affected prefixes to be probed.

LETsgo

@qps said:
Cogent's NOC/support is one of the best I've dealt with.

They actually have a pretty good NOC indeed. Responsive and technically skilled. However their company politics (congestion to some ISPs) are killing their reputation. They had a very bad rep five years ago and have recovered pretty good in the recent past.

strex

Are there some major ISPs with congestions to cogent at the moment?
Can someone name a few examples?

Wintereise

Comcast Communications would be in the #1 spot in that list, @strex.

strex

Comcast is an US ISP right?
I am running a download portal on servers which are connected only via cogent.
But I am only focusing EU users, and I have not received any complaints yet.

I wonder if there are some EU providers with congestions....

LETsgo

Dtag, France Telecom, etc

zfedora

Cogent <-> Comcast is complete crap in Chicago

CoreyM

crap in michigan aswell

c0y

Instead of bashing cogent for overloaded pipes you should maybe look at the ISPs who force all their traffic over Cogent... (especially the single homed ones)

FRCorey

Cogent does not prioritize pings on their routers.

staticsafe

This is a very good read BTW:

[PDF] A Practical Guide to (Correctly) Troubleshooting with Traceroute

doughmanes

@Frost said:
Instead of bashing cogent for overloaded pipes you should maybe look at the ISPs who force all their traffic over Cogent... (especially the single homed ones, or the greedy jew ones)

Like Comcast pushing 80-90%+ across TATA

qps

@LETsgo said:
They actually have a pretty good NOC indeed. Responsive and technically skilled. However their company politics (congestion to some ISPs) are killing their reputation. They had a very bad rep five years ago and have recovered pretty good in the recent past.

It's cheap bandwidth, and for many routes, it's great. As long as you go into it with that in mind, it makes a great addition to a multi-homed network blend.

It would be nice if they introduced some more BGP communities so it was easier to control incoming routes.

concerto49

People should stop bashing Cogent like the plague. It has its uses. Due to it being cheap, it's heavily used = there are a lot of direct routes from Cogent to Cogent. It means having it in the network is a good thing.

However should we rely on it alone? No.

Maounique

@concerto49 said:
However should we rely on anything alone? No.