New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
In today’s episode of “I will use big providers because they never have downtime. Small webhosting company Blyat”
Did anyone try turning it off and on? Power cycling usually fixes the Internet for me.
probably was bgp...
Swap out DNS for BGP and you've got yourself a new meme
I'm confused...why would Amazon Web Services use Fastly? They have their own CDN, CloudFront.
“ Amazon’s own retail website actually runs through Fastly, rather than CloudFront, and has done since May 2020.”
https://www.theguardian.com/technology/2021/jun/08/edge-cloud-error-tuesday-internet-outage-fastly-speed
AWS is even too expensive for Amazon lol, I can’t come up with another explanation. Reminds me of a time when an employee of a big tech company with a respected cloud service rented dedicated servers from me because their employee discount still costed 2x low end rates lol.
More POPs:
https://aws.amazon.com/about-aws/global-infrastructure/ vs
https://www.fastly.com/network-map
I suspect it was Chia.
Fastly is more advanced? And if your own cdn will go down what will you do?
And web site. Use Twitter... If it is not on AWS
So that is why Paypal wouldn't take my password? It shouldn't be related...
The Chia virus involucrated their servers
Translation: They incompetently committed two arch-sins:
I don't see anything like that in the post-mortem. In fact they specify the config was valid, just applying it triggered a bug previously introduced (long ago, in fact, so it was a very rare thing/combination of things, presumably).
@jsg nice throwing around the words "incompetent", but:
This seems like an edge case in a valid configuration change, how do you prevent that from happening with input validation.
PoPs are usually shared with event loop based software running, how do you perform isolation with such software? (Please don't say threads or per customer IPs. None of those scale for a CDN service.)
Run code that require isolation in eBPF or WebAssembly.
I can reach 100Gbps on six CPU cores using eBPF (and a lot of non eBPF code).
Also, my software is based on continuous polling, not event loops.
I accept that that may sound normal/OK to many. From an IT-security and proper system and software design POV however this sounds like a confession of bloody incompetence.
Now, one may discuss whether the bug is the problem or the fact that one customer "pushing a config" can bring down virtually the whole network or ... but to me it all says the same: too much marketing and large corp blabla and too little tech competence.
Note that I did not say "run away!" or "switch over to competitor XYZ!". Simple reason: while one may exist I do not know of any major CDN provider with sound engineering down to the core.
And yet I'm about to hear from you how you would prevent such a situation, apart from giving each customer their own thread, IP (or maybe containers), none of which actually scale for a CDN service.
If a certain engineering methodology is nearly impossible to put in practice, you can hardly claim incompetence.
By properly designing and engineering.
When they have bugs and when a bug can bring down virtually their whole operation, one can claim incompetence.
Btw, Stockholm syndrome? Or why do we have this discussion? Hello, earth to some users here: they f_cked up. Big time. And your position is "No, one must not call them incompetent!" and protecting them?
Sorry but contrary to, so it seems, popular believe bugs and shoddy designing and engineering are not somehow God-given and unavoidable.
jsg be like:
The end is nigh.
@stevewatson301
[self censored for the sake of politeness]
jsg be like:
Thanks a lot for confirming my sig.
Oh wow, I always thought Amazon uses their own CDN. Didn't know they are using Fastly as well
That's why there's QA, because non-zero bugs are unavoidable and QA is there to catch and prevent catastrophic events. This is a QA failure, not an engineering failure.
Did it work as designed or did it not? If it worked according to design but led to catastrophe, it was a design fault. If it didn't function according to design, it's QA's fault. The way it was described is a QA fail.