Github was hit by 1.35Tb DDoS attack establishing a new record
New on LowEndTalk? Please read our 'Community Rules' by clicking on it in the right menu!

Github was hit by 1.35Tb DDoS attack establishing a new record

This week GitHub was hit by 1.35Tb DDoS attack. It thus established a new record. After few minutes the attack was mitigated by Akamai, who was called in by GitHub.

Interestingly, the attackers used misconfigured Memcached servers to amplify the DDoS attack.
Memcached servers that are exposed to the world offer a huge attack multiplier - for each byte sent to them with a spoofed sender's address, you can expect a 51 kbyte response sent to the faked address. As a result, attackers can achieve the effect of 51,000 times more powerful than if they attacked the victim's server directly, further hiding their identity.
You can read more here: https://githubengineering.com/ddos-incident-report/
...and here: https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port-11211/

What are your thoughts? Will you update your Memcached configuration now?

Ultra Affordable VPSs - if you can find cheaper ones let me know! No downtime for nearly two years! | 6GB KVM VPS for $5,99/mo!
Hybrid Servers - create customizable VPSs on dedicated node managed by wishosting.com | 1TB KVM Storage VPS for $14,99/qt!

«1

Comments

  • omelasomelas Member

    How they measure its bandwidth? Surely they don't have 1tbps fiber to it?

  • jetchiragjetchirag Member
    edited March 2

    MrPsycho said: Will you update your Memcached configuration now?

    No choice there!

  • ClouviderClouvider Member, Provider

    @omelas said:
    How they measure its bandwidth? Surely they don't have 1tbps fiber to it?

    Erm, as per the graph on the border routers.

    Thanked by 1Aidan

    Clouvider Leading UK Cloud Hosting solution provider || UK Dedicated Servers Sale || Tasty KVM Slices || Latest LET Offer

    Web hosting in Cloud | SSD & SAS True Cloud VPS on OnApp | Private Cloud | Dedicated Servers | Colocation | Managed Services

  • sibapersibaper Member

    @omelas said:
    How they measure its bandwidth? Surely they don't have 1tbps fiber to it?

    from akamai? they switch to akamai when DDoS happen

    We only support unsupported OS!

  • MrPsychoMrPsycho Member

    omelas said: How they measure its bandwidth? Surely they don't have 1tbps fiber to it?

    Every single networking gear reports how many packets in/out it has transferred. Including their size. Later on, it was automatically summarized and presented on the graph.
    If you are curious if they have 1tbps "fiber", then yes. They do. By combining multiple links, they were able to receive 1.35Tb as per the graph. Don't compare it to your home internet - it doesn't work like that. It's not about what your ISP lets you use. It's about the gear that can handle enough packets. Internet isn't point-to-point. Packets go through lots of networks until they reach the final destination. As you can see on the graph Akamai was able to receive a total of 1.35Tb of data per sec.

    Thanked by 2Clouvider PandaRain

    Ultra Affordable VPSs - if you can find cheaper ones let me know! No downtime for nearly two years! | 6GB KVM VPS for $5,99/mo!
    Hybrid Servers - create customizable VPSs on dedicated node managed by wishosting.com | 1TB KVM Storage VPS for $14,99/qt!

  • stefemanstefeman Member
    edited March 2

    Why censor image?

  • MrPsychoMrPsycho Member

    stefeman said: Why censor image?

    Don't ask me, ask Github. They were the ones that censored it. I simply reuploaded image from their blog for imgur.com

    Ultra Affordable VPSs - if you can find cheaper ones let me know! No downtime for nearly two years! | 6GB KVM VPS for $5,99/mo!
    Hybrid Servers - create customizable VPSs on dedicated node managed by wishosting.com | 1TB KVM Storage VPS for $14,99/qt!

  • Attracting such a huge DDoS is something to be proud off right? I mean you must be doing something right for so many assholes to hate you!

  • MrPsychoMrPsycho Member

    Abdussamad said: Attracting such a huge DDoS is something to be proud off right? I mean you must be doing something right for so many assholes to hate you!

    I believe it was either a whitehat that tried to visualize the attack potential or a guy that wanted to test his capabilities.

    Ultra Affordable VPSs - if you can find cheaper ones let me know! No downtime for nearly two years! | 6GB KVM VPS for $5,99/mo!
    Hybrid Servers - create customizable VPSs on dedicated node managed by wishosting.com | 1TB KVM Storage VPS for $14,99/qt!

  • stefemanstefeman Member

    The funny thing is, this method is already utilized by booters.

  • ZerpyZerpy Member

    Mitigated bits being 1.3 Tbps - so the same as OVH just had - now we want the exact number from both to see who has the biggest ddos pe***.

  • 6ixth6ixth Member

    It was me.

  • MrPsychoMrPsycho Member

    6ixth said: It was me.

    U fund de wae to DDoS GitHub?

    Thanked by 1Wolveix

    Ultra Affordable VPSs - if you can find cheaper ones let me know! No downtime for nearly two years! | 6GB KVM VPS for $5,99/mo!
    Hybrid Servers - create customizable VPSs on dedicated node managed by wishosting.com | 1TB KVM Storage VPS for $14,99/qt!

  • graphicgraphic Member

    @MrPsycho said:

    6ixth said: It was me.

    U fund de wae to DDoS GitHub?

    What?

    End-to-end encrypted cloud storage, this link gives you 6gb free storage and me +1gb extra storage : sync.com

  • AlexBarakovAlexBarakov Member, Provider
    edited March 2

    Vastly exaggerated numbers.

    EDIT: Personal opinion.

    AlphaVPS - OpenVZ and KVM, DDoS Protected VPS in London, UK | Sofia, BG and NYC, US

  • TomTom Member, Host Rep

    @AlexBarakov said:
    Vastly exaggerated numbers.

    How come?

  • AidanAidan Member

    @AlexBarakov said:
    Vastly exaggerated numbers.

    How so?

  • bsdguybsdguy Member

    Just a sidenote: Those 0.05TB that were not mitigated are 50 Gb, not exactly a tiny almost nothing.

    Oh and, I suggest you take this kind of attack not as a monster-maximum but as the new normal, at least in terms of a trend. The 2 main ingredients are common enough, namely udp and asymmetric request/response size.

    Obviously both the memcached team and many (most?) users are utterly incompetent retards who ignored very basic rules, but one would be very mistaken to believe that that is an exception.

    I'm waiting for symantec to get more active in the DDOS protection snakeoil business...

    Thanked by 2default Ole_Juul

    My favourite prime number is 42. - \forall cpu in {intel, amd, arm}: cpu->speed -= cpu->speed/100 x irandom(15, 30) | state := hacked

  • We are getting soon close to breaking the internet if the target is not a anycast network. Bets when it will happen?

  • WHTWHT Member

    And why someone will ddos github?

  • MrPsychoMrPsycho Member

    @WHT said:
    And why someone will ddos github?

    Why not?

    Ultra Affordable VPSs - if you can find cheaper ones let me know! No downtime for nearly two years! | 6GB KVM VPS for $5,99/mo!
    Hybrid Servers - create customizable VPSs on dedicated node managed by wishosting.com | 1TB KVM Storage VPS for $14,99/qt!

  • FHRFHR Member, Provider

    @AlexBarakov said:
    Vastly exaggerated numbers.

    This incident was discussed on NANOG mailing list, those are not exaggerated numbers, this was real traffic. Server and network providers like Digital Ocean and NTT took measures to limit the impact - e.g. NTT started to rate-limit udp/11211 (memcached) on all their external interfaces.

    It seems like memcached has an amplification factor of around 50000. And it's not that hard to scan the whole internet and find unfirewalled memcached servers.

    Thanked by 1PandaRain

    SkylonHost | Affordable Semi-Dedicated VPS - Enjoy performance to the fullest extent. | 40% OFF promo

  • FHRFHR Member, Provider

    @WHT said:
    And why someone will ddos github?

    Someone probably wanted to test firepower of their newest booter. So they chose some target who they knew would mitigate the attack and publicly share numbers.

    SkylonHost | Affordable Semi-Dedicated VPS - Enjoy performance to the fullest extent. | 40% OFF promo

  • AlexBarakovAlexBarakov Member, Provider

    @FHR said:

    @AlexBarakov said:
    Vastly exaggerated numbers.

    This incident was discussed on NANOG mailing list, those are not exaggerated numbers, this was real traffic. Server and network providers like Digital Ocean and NTT took measures to limit the impact - e.g. NTT started to rate-limit udp/11211 (memcached) on all their external interfaces.

    It seems like memcached has an amplification factor of around 50000. And it's not that hard to scan the whole internet and find unfirewalled memcached servers.

    While it was only a personal opinion, I can confirm that any sane provider I work for or work with has taken measures to limit the impact.

    AlphaVPS - OpenVZ and KVM, DDoS Protected VPS in London, UK | Sofia, BG and NYC, US

  • again... like in 2014... Fuck, I really hate these exploits... Btw, I heard few days ago about new exploit related to torrent network and ddos attacks organization by that (I don't talk about leaked exploit with uTorrent and RSS), I talk about something a little bit different...

  • RhysRhys Member

    AlexBarakov said: EDIT: Personal opinion.

    cringe

  • bsdguybsdguy Member

    @Rhys said:

    AlexBarakov said: EDIT: Personal opinion.

    cringe

    We can be really glad to have such valuable, well spoken and insightful commenters like you here ...

    @AlexBarakov

    Could you tell us a bit more about your point of view ("Vastly exaggerated numbers.")?
    I was a bit bewildered, too, when I read that but I guess you have a reason for your opinion and I'd be interested in it.

    My favourite prime number is 42. - \forall cpu in {intel, amd, arm}: cpu->speed -= cpu->speed/100 x irandom(15, 30) | state := hacked

  • adxnadxn Member, Provider

    @6ixth said:
    It was me.

    Sincerely,

    Shubhankar From Hyperpage

  • AppleApple Member

    @WHT said:
    And why someone will ddos github?

    Sadly people attack anything these days, they do not need a reason.

    UK Based Anti DDOS VPS - v2internet.com // EU Anti DDOS VPS - HostingAndVPS.com //
    cPanel Server Management - cPanelServerManagement.co.uk

  • WebGuruWebGuru Member

    Oles posted this list on his twitter. OVH was also hit by 1.3tb/s and it looks like most IPs/ASN are from China. (No surprise i also get hit by chinese ip addresses every day)

    Thanked by 1netomx
  • ClouviderClouvider Member, Provider

    Pity the GFW is not used to prevent from actual attacks.

    Clouvider Leading UK Cloud Hosting solution provider || UK Dedicated Servers Sale || Tasty KVM Slices || Latest LET Offer

    Web hosting in Cloud | SSD & SAS True Cloud VPS on OnApp | Private Cloud | Dedicated Servers | Colocation | Managed Services

  • SplitIceSplitIce Member, Provider
    edited March 3

    @Neoon said:
    Wasn't the Mirai botnet 1.6Gbit?

    I don't believe Mirai broke 1Tbps. We saw around 300Gbps (from memory) when we got hit. By the end though it dropped as multiple targets were being hit by the same devices reducing the traffic hitting each target.

    X4B - DDoS Protection: EU & US affordable DDoS protection including Layer 7 mitigation.
    Latest Offer: 1TB and 2TB Anycast DDoS Protection (March Madness)
  • qtwrkqtwrk Member

    @Clouvider said:
    Pity the GFW is not used to prevent from actual attacks.

    lol, it's probably gonna have white-list mode in stead of current black-list mode, I guess you wouldn't be bothered by hit by Chinese IP anymore by then

    netcup 5 euro coupon: 36nc15324722143 36nc15324722144

  • edited March 3

    @Apple said:

    @WHT said:
    And why someone will ddos github?

    Sadly people attack anything these days, they do not need a reason.

    I've never understood the logic either. It seems to me like it's mostly just script kiddies who just get a kick out of it. Like it's a video game or something. Maybe for bragging rights.

  • jackbjackb Member, Provider
    edited March 3

    @LosPollosHermanos said:

    @Apple said:

    @WHT said:
    And why someone will ddos github?

    Sadly people attack anything these days, they do not need a reason.

    I've never understood the logic either. It seems to me like it's mostly just script kiddies who just get a kick out of it. Like it's a video game or something. Maybe for bragging rights.

    It is common for criminals and script kiddies to prove their DDoS services by taking down high profile & well protected sites. Krebs' site was a very notable example in the past.

    Afterburst - Awesome OpenVZ&KVM VPS in US+EU

  • risharderisharde Member

    Wow, that is some serious throughput figures! Thank goodness my memcache server was behind nat.

  • ZerpyZerpy Member

    @risharde said:
    Wow, that is some serious throughput figures! Thank goodness my memcache server was behind nat.

    Sane people do basic firewalls.. right?

  • RafayRafay Member

    OMG!
    who whould like to compare OVH vs Github protection now?
    OVH also get about 1.3TB DDOS attack so i think they both have pretty much great protection.

  • corbpiecorbpie Member

    Back in the day my minecraft server withstood a 512mb UDP flood for 1 hour

    grape

  • Ole_JuulOle_Juul Member

    Clouvider said: Pity the GFW is not used to prevent from actual attacks.

    Are you suggesting that there is a trade deficit with network packets.?

  • hzrhzr Member

    bsdguy said: Obviously both the memcached team and many (most?) users are utterly incompetent retards who ignored very basic rules, but one would be very mistaken to believe that that is an exception.

    Effectively every single distro-shipped/packaged version binds to localhost only. The only incompetent retards involved here are people either running it in docker and intentionally forwarding 0.0.0.0:* instead of appropriately linking containers, or people compiling from scratch without reading the readme and without reading the multiple comments about it.

    Thanked by 3Aidan Sumeragi FHR
  • SplitIceSplitIce Member, Provider

    @hzr said:

    bsdguy said: Obviously both the memcached team and many (most?) users are utterly incompetent retards who ignored very basic rules, but one would be very mistaken to believe that that is an exception.

    Effectively every single distro-shipped/packaged version binds to localhost only. The only incompetent retards involved here are people either running it in docker and intentionally forwarding 0.0.0.0:* instead of appropriately linking containers, or people compiling from scratch without reading the readme and without reading the multiple comments about it.

    And people running popular pre-packaged software that includes an incorrectly configured Memcached e.g Zimbra

    Thanked by 3pike Aidan FHR
    X4B - DDoS Protection: EU & US affordable DDoS protection including Layer 7 mitigation.
    Latest Offer: 1TB and 2TB Anycast DDoS Protection (March Madness)
  • edited March 4

    @SplitIce said:

    @hzr said:

    bsdguy said: Obviously both the memcached team and many (most?) users are utterly incompetent retards who ignored very basic rules, but one would be very mistaken to believe that that is an exception.

    Effectively every single distro-shipped/packaged version binds to localhost only. The only incompetent retards involved here are people either running it in docker and intentionally forwarding 0.0.0.0:* instead of appropriately linking containers, or people compiling from scratch without reading the readme and without reading the multiple comments about it.

    And people running popular pre-packaged software that includes an incorrectly configured Memcached e.g Zimbra

    CentOS v7 (and I presume v6 and v5) memcached binds to 0.0.0.0 by default. Luckily the fix is simple.

    I'm sure that RPM will be updated shortly.

  • bsdguybsdguy Member
    edited March 4

    @hzr said:

    bsdguy said: Obviously both the memcached team and many (most?) users are utterly incompetent retards who ignored very basic rules, but one would be very mistaken to believe that that is an exception.

    Effectively every single distro-shipped/packaged version binds to localhost only. The only incompetent retards involved here are people either running it in docker and intentionally forwarding 0.0.0.0:* instead of appropriately linking containers, or people compiling from scratch without reading the readme and without reading the multiple comments about it.

    I'd say that if something serves as the basis for one of the top-5 DDOS ever "but it's the users fault!" doesn't cut it anymore.

    Well noted, I understand perfectly well that udp has its attractive sides but we all also know fucking well that udp by its very nature suffers from a spoofed source problem. That, plus the fact that something, in this cache memcached, can by design be used as a major amplifier/reflector would tell everyone with a brain to put some safety into his stuff. And no, "the default config is OK" is not some safety but a lame excuse.

    You see, we have pretty much wire speed sym crypto and plenty of other funny devices available nowadays (well, actually since many years). Hell, even shitty open/libre/polar/snakeoil-ssl/tls are good enough for pretty high post-setup speeds (e.g. aes). Even better, I'm not even asking for encryption but merely for some kind of access control, say, one (1) single fucking TCP session setup packet exchange.

    So no, there is no fucking excuse for memcached acting like an ignorant lobotomized retard!

    My favourite prime number is 42. - \forall cpu in {intel, amd, arm}: cpu->speed -= cpu->speed/100 x irandom(15, 30) | state := hacked

  • hzrhzr Member

    LosPollosHermanos said: CentOS v7 (and I presume v6 and v5) memcached binds to 0.0.0.0 by default. Luckily the fix is simple.

    >

    CentOS also firewalls all ports by default, doesn't it?

  • hzrhzr Member

    bsdguy said: one (1) single fucking TCP session setup packet exchange

    I like how QUIC handles this issue

  • eva2000eva2000 Member
    edited March 4

    LosPollosHermanos said: CentOS v7 (and I presume v6 and v5) memcached binds to 0.0.0.0 by default. Luckily the fix is simple.

    Don't think i ever seen that usually defaults to localhost or 127.0.0.1 and on centos 7 firewalld blocks UDP by default https://access.redhat.com/solutions/3369081 and https://bugzilla.redhat.com/show_bug.cgi?id=1551182

    Not 100% sure though I always had source compiled memcached with 127.0.0.1 default and CSF Firewall enabled blocking the ports.

    Thanked by 1Aidan
    * Centmin Mod Project (HTTP/2 support + ngx_pagespeed + Nginx Lua + Vhost Stats)
    * Centmin Mod LEMP Stack Quick Install Guide
  • @eva2000 said:
    Don't think i ever seen that usually defaults to localhost or 127.0.0.1 and on centos 7 firewalld blocks UDP by default https://access.redhat.com/solutions/3369081 and https://bugzilla.redhat.com/show_bug.cgi?id=1551182

    Not 100% sure though I always had source compiled memcached with 127.0.0.1 default and CSF Firewall enabled blocking the ports.

    It's bound to all interfaces and turned off when installed. You enable the service and remove the firewall restrictions and it's open to the net. Memcached compiled by default is restricted to 127.0.0.1 for UDP, but not in the repos of major Distros, which is part of the issue.

    Thanked by 2hostdare Aidan
Sign In or Register to comment.