Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Hetzner DDoS issues in Production
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Hetzner DDoS issues in Production

mustafammustafam Member
edited July 2021 in General

Considering Hetzner dedicated, for a non-gaming B2B solution. Need perhaps 100+ production servers. I have a major concern about Hetzner DDoS mitigation, which uses Arbor Networks hardware. I believe it kicks in after 2-5 minutes, which is fine. I can handle application layer attacks via nginx and some custom method. My problem is: DDoS seems to mess up the SSL handshake.

I have not experience this myself, but I have found three threads talking about this. One person said that moving to OVH solved this problem. But I do not like OVH.

Does anybody have first-hand knowledge of this? Was there a solution. I do not use a third-party service like Cloudflare, nor do I want to.

Also, are there other major problems with using Hetzner for a production solution?

Thanks!

«1

Comments

  • LordSpockLordSpock Member, Host Rep

    Hetzner's DDoS protection is adequate but by no means perfect.

    I've not experienced these problems myself - but I have heard of a few people who have had issues with it.

    If you are in such a large scale in production - it would be better to look for someone who would take the time to make sure their solution works for you.

    Thanked by 1mustafam
  • jordynegen11jordynegen11 Member
    edited July 2021

    I've got servers at both OVH and Hetzner.

    DDOS protection at hetzner is not great. If very easy to even down their 10Gbit servers with a high volume attack. However the price/performance is great and their support is fast.

    OVH has way better protection but their support is the worst I have ever seen. Both have their compromises.

  • DPDP Administrator, The Domain Guy

    If you have no issues with using your phone to call OVH for support-related matters, then you're good to go.

  • @LordSpock It is concerning that others have had this issue. We're gonna simulate a DDoS on any server before purchasing. Are we allowed to DDoS ourselves on Hetzner (or any other host for that matter) for testing purposes?

  • LordSpockLordSpock Member, Host Rep

    @mustafam said:
    @LordSpock It is concerning that others have had this issue. We're gonna simulate a DDoS on any server before purchasing. Are we allowed to DDoS ourselves on Hetzner (or any other host for that matter) for testing purposes?

    You aren't without direct permission from everyone involved in that chain.

    Thanked by 1pike
  • deankdeank Member, Troll

    Someone's going to burn the bridge with Hetzner.

    Let him do it.

    Thanked by 2yoursunny vimalware
  • @jordynegen11 I truly loathe OVH. I wanted to like it. Even their admin is terriblly slow. It feels like 1999 dialup. Does OVH have an other show-stopper problems, besides terrible support and slow admin? Do you see down-times? One potential idea is to go with OVH with multiple redundant servers. I really wanted to go with Hetzner, becauses it's so slick. But the SSL DDoS issue is truly a show-stopper.

  • @LordSpock said:

    @mustafam said:
    @LordSpock It is concerning that others have had this issue. We're gonna simulate a DDoS on any server before purchasing. Are we allowed to DDoS ourselves on Hetzner (or any other host for that matter) for testing purposes?

    You aren't without direct permission from everyone involved in that chain.

    Thanks for the heads up. Do I contact support and ask them: "We need 100 servers, can we run DDoS tests?". Does that sound right, or is it laughable?

  • deankdeank Member, Troll

    Laughable.

    Thanked by 3mustafam MrH pike
  • yoursunnyyoursunny Member, IPv6 Advocate

    @mustafam said:
    Does OVH have an other show-stopper problems, besides terrible support and slow admin? Do you see down-times?

    Yes, OVH is literally on fire.
    https://yoursunny.com/t/2021/OVH-halt-and-catch-fire/

  • mustafammustafam Member
    edited July 2021

    Yeah, I know the fire issue. But to their defense, that was truly a one-off thing. And, I can plan for this with servers spread out in 2 of their data centers.

  • @mustafam said:

    @LordSpock said:

    @mustafam said:
    @LordSpock It is concerning that others have had this issue. We're gonna simulate a DDoS on any server before purchasing. Are we allowed to DDoS ourselves on Hetzner (or any other host for that matter) for testing purposes?

    You aren't without direct permission from everyone involved in that chain.

    Thanks for the heads up. Do I contact support and ask them: "We need 100 servers, can we run DDoS tests?". Does that sound right, or is it laughable?

    Why not tell them your use case and concerns?

  • jackbjackb Member, Host Rep
    edited July 2021

    @mustafam said:

    @LordSpock said:

    @mustafam said:
    @LordSpock It is concerning that others have had this issue. We're gonna simulate a DDoS on any server before purchasing. Are we allowed to DDoS ourselves on Hetzner (or any other host for that matter) for testing purposes?

    You aren't without direct permission from everyone involved in that chain.

    Thanks for the heads up. Do I contact support and ask them: "We need 100 servers, can we run DDoS tests?". Does that sound right, or is it laughable?

    "Hi there. I'd like to buy some stuff from you, but first let me shit in a bag on your doorstep and set it on fire. Is that OK?"

  • @mustafam said:
    Considering Hetzner dedicated. Need perhaps 100+ production servers. I have a major concern about Hetzner DDoS mitigation, which uses Arbor Networks hardware. I believe it kicks in after 2-5 minutes, which is fine. I can handle application layer attacks via nginx and some custom method. My problem is: DDoS seems to mess up the SSL handshake.

    I have not experience this myself, but I have found three threads talking about this. One person said that moving to OVH solved this problem. But I do not like OVH.

    Does anybody have first-hand knowledge of this? Was there a solution. I do not use a third-party service like Cloudflare, nor do I want to.

    Also, are there other major problems with using Hetzner for a production solution?

    Thanks!

    Why not actually call them and get a real live account manager. Even at the cheapest if you are looking for Dedis thats at least 4500 a month for the 100 servers. Plus who knows you might even get a volume discount and other perks - assuming this is all legit and not theoretical

  • @databoss said:

    @mustafam said:
    Considering Hetzner dedicated. Need perhaps 100+ production servers. I have a major concern about Hetzner DDoS mitigation, which uses Arbor Networks hardware. I believe it kicks in after 2-5 minutes, which is fine. I can handle application layer attacks via nginx and some custom method. My problem is: DDoS seems to mess up the SSL handshake.

    I have not experience this myself, but I have found three threads talking about this. One person said that moving to OVH solved this problem. But I do not like OVH.

    Does anybody have first-hand knowledge of this? Was there a solution. I do not use a third-party service like Cloudflare, nor do I want to.

    Also, are there other major problems with using Hetzner for a production solution?

    Thanks!

    Why not actually call them and get a real live account manager. Even at the cheapest if you are looking for Dedis thats at least 4500 a month for the 100 servers. Plus who knows you might even get a volume discount and other perks - assuming this is all legit and not theoretical

    They even have a custom solutions page these days. https://www.hetzner.com/custom-solutions

  • Tr33nTr33n Member
    edited July 2021

    @mustafam said: My problem is: DDoS seems to mess up the SSL handshake.

    What you mean is probably that the first connection is rejected. That happens due to TCP syn auth to distinguish between legitimate and spoofed syn packets. A legitimate client normally attempts a reconnect, then the connection is allowed.

    You could ask Hetzner if they can turn off the TCP syn protection for you. But I guess, they don't make adjustments.

  • @databoss said:
    Why not actually call them and get a real live account manager. Even at the cheapest if you are looking for Dedis thats at least 4500 a month for the 100 servers. Plus who knows you might even get a volume discount and other perks - assuming this is all legit and not theoretical

    Absolutely will. But it is savvy to get an independent review. They will likely play down this issue of DDoS SSL failures to get the sale. So, I don't want to contact them, until I've done probing myself.

  • databossdataboss Member
    edited July 2021

    @mustafam Or they may actually help you test it, you know do some engineering and make a client happy. A vendor is a partner- not an adversary- unless you are going to turn and burn out of there after 30 days, or not hit any agreed commitments

    Thanked by 1mustafam
  • jarjar Patron Provider, Top Host, Veteran
    edited July 2021

    I'm not running game servers or any kind of UDP data stream that suffers from latency spikes. However, my usage does involve a large number of frequent and small connections (email). Hetzner DDOS protection has been more than adequate. I've had a few short outages from a DDOS but I've had them everywhere, no network excluded. You're going to have them everywhere unless you just don't piss anybody off (or end up on the other side of a dice roll). If the perfect DDOS protection existed and could be shared on a reasonable budget, the methods used to perform DDOS attacks would adjust.

  • @mustafam said:

    @LordSpock said:

    @mustafam said:
    @LordSpock It is concerning that others have had this issue. We're gonna simulate a DDoS on any server before purchasing. Are we allowed to DDoS ourselves on Hetzner (or any other host for that matter) for testing purposes?

    You aren't without direct permission from everyone involved in that chain.

    Thanks for the heads up. Do I contact support and ask them: "We need 100 servers, can we run DDoS tests?". Does that sound right, or is it laughable?

    It would be a red flag that you're a shitty customer.

    Contact sales, explain your current issues and how you'd like to stress it to know if it works better.

    Are you English as a second language and can't be bothered to write a paragraph about who you are and what you need? I think someone more responsible than you should do the talking.

  • @TimboJones said:
    Are you English as a second language and can't be bothered to write a paragraph about who you are and what you need? I think someone more responsible than you should do the talking.

    Obviously, when I do inquire, I would phrase it professionally.

  • KassemKassem Member

    Congrats on having a product big enough to need 100 dedicated servers!

    Shouldn't these 100 servers be divided across different data centers and providers instead of 100 servers with just Hetzner? Like 25-25-25-25 so 4 different dcs and providers.

    Is it cheaper to run it on dedicated servers and worrying about faulty hardware vs distributing it across different regions on a public cloud or clouds?

    Any reasons for totally avoiding Cloudflare? I know some people don't like it but they do this stuff all day.

    As yoursunny mentioned, OVH just recently conducted major cloud migration which should rule it out of any production stuff.

  • mustafammustafam Member
    edited July 2021

    @kassem thanks for the congrats!

    We are planning to distribute in 2 Hetzner regions.

    About VPS: we need a ton of bandwidth, plus we don't like noisy neighbors. We're not storing data locally, so we couldn't care about faulty drives.

    About Cloudflare: can't use because of the dynamic nature of our apps. Also, adds one more moving part. But, we'll do a re-evaluation.

    About OVH: it was just one data center on fire. And we were thinking of their Montreal data center, which I believe is fairly stable. But again, OVH is the the very last choice. Their support + admin are super slow.

  • jordynegen11jordynegen11 Member
    edited July 2021

    @mustafam said:
    @jordynegen11 I truly loathe OVH. I wanted to like it. Even their admin is terriblly slow. It feels like 1999 dialup. Does OVH have an other show-stopper problems, besides terrible support and slow admin? Do you see down-times? One potential idea is to go with OVH with multiple redundant servers. I really wanted to go with Hetzner, becauses it's so slick. But the SSL DDoS issue is truly a show-stopper.

    I understand that. Support is terrible and as you can read here, the Ipv6 network of OVH is a drama.

    But I never experience any downtime besides some motherboard failures. The monitor service of OVH is actually pretty good. Most of the time faulty components will be replaced within 2 hours.

    You can also create a GRE or wireguard tunnel between OVH and hetzner and route DDOS protected OVH IP's to your hetzner servers B). We ectually use that for our budget game hosting. but that means more points of possible failure.

    Thanked by 1mustafam
  • pikepike Veteran
    edited July 2021

    @mustafam said:

    Yeah, I know the fire issue. But to their defense, that was truly a one-off thing. And, I can plan for this with servers spread out in 2 of their data centers.

    How can you say that with certainity? The other datacenters most likely dont have better protection against such incidents. Remember their "datacenter" basically was a four level shed with wooden floors and no fire prevention at all. Let alone the "datacenter" right next to that which was basically made of shipping containers. When the fire brigade arrived they could do nothing but try to let the building burn down in a controlled way.

    https://www.ovh.de/images/news/rbx4/datacentre_rbx4_ensemble.jpg

    If you look at Hetzner datacenters for example, if a fire occurs there it wouldnt spread to the whole datacenter quickly (as it has only one level, and fire-proof material), so the damage could be minimized by the fire brigade, as they can access the burning parts and separate them from the still intact ones.

    https://www.hetzner.com/de/assets/Uploads/unternehmen/datacenterpark-fsn.jpg

  • @jordynegen11 said:

    But I never experience any downtime besides some motherboard failures.

    You, know that's actually very good to know. I've read similar comments on HackerNews and LowEndTalk. Baically, support + admin are terrible at OVH, but uptime is rock-solid.

  • mustafammustafam Member
    edited July 2021

    @pike said:

    How can you say that with certainity? The other datacenters most likely dont have better protection against such incidents.

    Good point. The thing is, it doesn't matter. We're gonna have redundant boxes in a separate data center. We'll switch to those in case of a fire in the main data center. It will act like spare tire. And, any DB data is replicated anyways via point-in-time-recovery; uploaded to durable storage: S3 or OVH object storage.

    If a fire happens in 2 data centers at the same time, no problem still. With our one-click DB recovery, we'll just shift to a different data center or at this point, probably a different provider.

  • @mustafam The other thing is- is your credit good enough, have you been in business long enough for them to commit 5 grand a month in hardware to you - with post pay billing? or can you prepay?

  • @databoss

    Prepay. The cost is peanuts compared to AWS. Surprisingly, it's less than an employee's salary.

  • It's really hard to find a provider that would cater to both hardware and specialized DDoS protection needs. Perhaps some remote protection via Path would work? They have L7/HTTP filters as well prob have some solutions for the "dynamic nature of [your] apps". Or perhaps Magic Transit by Cloudflare, but it wont be cheap.

Sign In or Register to comment.