New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
This hurt to read:
Really cool of them to be so detailed in this though.
#justhurricaneelectricthings
Indeed, with all things like this it's what they did and are doing about it that sets them apart from so many others.
These detailed analysis can really help out hundreds of existing hosting/cloud providers. It's better to be prepared than getting affected.
I have been a fan of Linode for a lot of years, March 10th 2008 to be precise. This situation however lost them some good customers to DigitalOcean over this. The ability to move a snapshot to a different unaffected region was cited many times.
They need to catch up and stay up, they tend to be on the back foot. An example is SSD, Chris said back in 2013 that SSD's were not a priority for Linode as they were happy with the performance of their set up of the time. No sooner than DO started to draw many clients away with their SSD offers did Linode then jump up and take notice. However what they offered in terms of quality was better than DO.
DO has the better features, Linode has the better infrastructure. In my view.
I always see you praising Linode, are you part of them?
As I said.
Customer for many years. If you want to attempt some drama from nowhere because I regularly praise a provider I use, trust and heavily rely on then go for it.
Seems to me like that's actually what they've been lacking is infrastructure. They've been relying on equipment and contracts with colocation providers instead of direct contracts with upstream and their own networking equipment, at least that's how I interpret it.
But it's good that they recognize that bottleneck and are taking steps to change it. I've always loved Linode myself and I'm directly avoiding the comparison to DO because it's totally possible to work for one and love both
Lee i respect you so i never would try to stir up anything. no need to be so defensive.
I guess I am including everything down to the hardware, however agreed. I guess I would be interested in how DO would handle the same situation, maybe they have and it's had a much less effect.
However like I said Linode are too often on the back foot reacting to things than planning for what is really the inevitable for all providers at some point.
Fair enough, apologies. I would never praise a provider I have involvement in, there have been discussions on here in the past about providers I am involved in and many on WHT. I never go near any of them.
Good to know.
...
...
Good
Let's see in the future.
Agreed. Their detailed explanation was a great start though.
Shouldn't IP address of router should be a none routeable IP?
Been amateur hour it seems.
From the post it mentions they relied on upstream transit from their colo providers (they don't roll their own network) and don't use carrier neutral locations like CoreSite, rookie mistake and you hate to see it.
Also seems like a cop out to blame upstream transit capacity, when they're selling that capacity downstream and advertising such.
Third, routers and infra were directly accessible, no VPN?
This is what happens when you don't run your own network. They relied on the ASN of other providers, so they couldn't quickly announce elsewhere.
If they used their own ASN in the US, they could have hopped back online with immediate back-hauling from a protected network. Instead they were at the mercy of upstream colocation providers who were getting knocked out because of Linode on a holiday.
Result? They were gone from the routing table likely waiting on a senior engineer to decide to remove the null-route.
Used to think of this company as the 'go to' stable company, but it's been built on a house of cards.
About time they signed some transit contracts.
There has been a lot of issues on Linode's infrastructure like this. Just under-dimensioned crazily.
The little schematic they put up of their proposed architecture will just lead to more headaches. They don't and never have had the skillset internally to build a network, they seem to believe that server admins == network architects.
Cisco products they've mentioned will be a disaster, they've not done the performance testing themselves, just read some Tolly 'report' which says the sponsor of the report is always wonderful.
I am continually testing all the major vendors routing/switching solutions, I wouldn't choose those products in this application.
Are there any LEB providers that actually do this correctly? (Genuine question.)
Yes, a ton of providers have an ASN + their own IPs. Not only for the reasons mentioned, but with the IP shortage over the last few years, everyone and their uncle seems to have a Delaware LLC with some IPs.
Also legimate providers have their own AS + transit.
Linode was just very lazy networking wise, I remember wondering why they had their own ASN but only used it in a handful of EU locations.
If they were smart they would have been on their own ASN in all US locations so they would be able to actively engineer their traffic and augment providers in real-time.
Instead they were at the mercy of angry higher-up networking guys at their upstream who were on holiday, and saw Linode the cause of other clients having issues.
Just like Linode and others will null-route your IP when you get atttacked, that happened up level, to them.
They were the problem child of transit in their data centers, and they didn't pay or bark loud enough to get the needful done on Christmas.
I know as I had to move everything out of Newark, NJ on Christmas with off-site backups and changed DNS... low TTL FTW
Poor planning, poor logistics and lack of proper infrastructure. Worse, from their postings, their equipment was publicly accessible without a VPN connection... which makes me face-palm even worse.
They can blame their upstreams all they'd like, but they failed hard in this, and it was their multiple hack, first time or two was the Bitcoin heist.
They're swiss cheese IMO until they hire someone competent to audit everything, I'm sure there's APT somewhere in there, and I'm not sticking around for it to be used.
We do in our US locations.
As a Linode user, it was rough watching this happen.
I'm afraid it seems to be quite common in the industry most capacity is oversold. Depending on how good the provider is will depending by how much.