New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Hetzner cloud in falkenstein down
Not affected myself. Just noticed because a couple services i use are down. I normally like Hetzner a lot but jesus that looks clueless: https://www.hetzner-status.de/en.html
Edit: Lol. The second it's posted at lowend talk the issue gets fixed. Massive importance or just coincidence? You decide
Comments
Wasn't expecting it to be offline tbh...
Humph, mine were off too. This has been happening alot recently
Yeah, i was also quite surprised when i saw that all the down servers belonged to Hetzner. Checked their status and was like: Ouch...
Really? I guess i'll be holding on to my old non cloud VPS a bit longer then.
Mine is up and shows 22 days uptime. Do you mean there was a network interruption? Any idea how long it lasted? My guess is other Hetzner VPS were also affected. No idea about dedis.
Yep and yep. But shit happens, and happens with anyone so let's just call it a day.
Though lately I've been noticing a few network hiccups.
Network interruption, https://www.hetzner-status.de/en.html
Yeah, seems there was some flaky router. According to the status page it started acting up around 12AM (GMT+1?) with numerous tries to repair it until about 1:00PM. I guess there were multiple shorter downtimes during the night/morning (must have been up sometimes judging by the side effects on the affected services). Anyways, at least between 12PM and 1:00PM 2 services i use were completely offline. My stuff in Nuremberg was unaffected anyways.
Interesting. Any idea if Nuremberg-Falkenstein connectivity was affected? One thing I like about Hetzner Cloud is you can easily have servers in both locations (plus Finland for that matter), with DNS round robin (I guess IP failover won't work if FLK is split from NBG)
Hey @Hetzner_OL how about offering anycast routing between the 2 (or 3) locations?
Yeah, it was. That's actually how i noticed it. I have a VPS in Nuremberg that tried to connect one of the affected services but failed. Traceroute timed out. IP belonged to Hetzner. Shrugged it off at first (non profit services go down every day...) but when i couldn't reach a couple websites and all of them also resolved to Hetzner i got suspicious.
My mysql replica experienced a 41-minute delay in replication, so I'd expect downtime to be roughly 41 minutes :-D
On the bright side, no impact on prod traffic
It still looks like there are issues (warning as soon as you login to the panel about the router). Is Hetzner to be trusted with large client websites? Got a few high-end clients I was thinking of migrating to the cloud panel but are they stable enough?
According to the status page the issue is fixed sine 1PM local time. If they are still warning people about potential issues i'd call that being pretty open about it. Their communication is always excellent anyways. Even if the issue handling looks a bit messy this time hats of for having the balls to post it as is. Other providers would simply stay quiet and sit it out.
On stability: While i haven't used their cloud line i am with them for about 10 years now and haven't experienced a single noteworthy downtime yet. Actually i was connected to one of their VPS almost 24/7 for a rather long time. During which i maybe witnessed about 2-3 <5m network hickups in a year and i am not even sure if the reason might have been on my side. I also have a VPS with around a year of uptime now if that says anything. For dedis you can pretty much assume they only reboot if you request it or there is a hardware failure (rare).
The dedis from Hetzner are solid, these cloud fuckups are also common for example at OVH from time to time.
General rule, the more stuff you add together, the more complex and fragile it will get.
Every time I tell people to get a dedi, even if its just a small one, its far better then any of the cloud "shit" or vps, whatever you wanna call it.
Hetzner Cloud is still new and maybe a little bit iffy based on this thread and another recent one. But, there is no server or service that will never have an outage. If you have low tolerance for downtime you have to set up a high availability solution with multiple servers.
I'd be interested to know if this outage only affected the cloud product, or also the regular VPS and dedicated servers.
Also regarding high-end clients: this is low end talk, which might not be such a good match for high-end anything. I've been using Hetzner stuff happily, but I'm a low end user and if my stuff is down I either don't notice or I'm at worst slightly inconvenienced. I don't have anything on Hetzner where a even a much longer temporary outage would cause me any real difficulty.
Also regarding high-end clients: other people are seeing better network speeds than me, but from my own experience I'd say if you have a lot of transatlantic traffic, Hetzner isn't the best possible choice. Their network within EU is much better than their transatlantic routes. I'm in the US and use Hetzner anyway because the hardware value is good enough for the trade-off to be worth it, but I do have to manage my expectations about network performance. If my requirements were more stringent I'd have to pay more and look for better connectivity.
Regarding the dedis: I've had one about 3 years and it's been very solid, zero crashes and just one unexpected downtime due to a hardware failure (bad cpu fan). This was not a crash because I shut down the server on purpose, but it was unexpected downtime because the server didn't come back up when I expected it to. I opened a ticket and Hetzner staff found and fixed the problem, all within about 30 minutes, which I thought was great.
Offline again
It's only a couple minutes (15 or so?) at a time but gets very annoying e.g. IRC connections dropping
I have one cloud node offline now, yesterday all my nodes and dedis were down.
@Tom @rick2610 is this still ongoing? At least right now the services i use that were affected yesterday seem to be online.
Nah, down for about ~45 mins
So you are up again?
For now, ya
Hi LET readers, Please see: https://www.hetzner-status.com/#9223 and https://www.hetzner-status.com/#9237 We will continue to post updates on our Hetzner Status site. And I will post more information when I can.
@willie "Hey @Hetzner_OL how about offering anycast routing between the 2 (or 3) locations?" -- Please give me a few days to ask my colleagues about this. --Katie, Marketing
@Hetzner_OL thanks for the update. I see the status page says "We are currently experiencing a fault on our router fsn1-dc8-ex9k2." Does that mean only DC8 is affected? If yes, maybe that leaked info about where the cloud servers are . Actual reason for asking is that some of us have dedis in other DC's. I think mine is in DC11.
I only saw a 15 min network blip on my HTTP reverseproxy VM on a Skylake dedi in FSN-DC1 (around 0712-0727 Hrs UTC.)
I don't have any cloud instances; so I guess I missed that outage.
It has always been known where the cloud servers are - the location in their panel literally states "fsn1-dc8"
@willie - I am unaware of any issues in FSN1-DC11 . I would write a support request via Robot. Please do this any time that you are having connectivity issues that have not yet been posted on Hetzner Status.
@vimalware - Ditto for FSN1-DC1.
--Katie, Marketing
I haven't had any trouble with my dedi that I know of. I just wondered if the network prob was isolated to DC8, but it sounds like it wasn't. My dedi is mostly a compute and storage box so if it has occasional network outages I generally won't notice.
The cloud in Falkenstein was again down for about 20 min today. I am a customer of Hetzner for 6 years but i never experienced so many problems as the last two weeks.
Wait till the end of the World Cup