New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Memcrashed - Major amplification attacks from UDP port 11211
fix your memcached server people
Comments
All those running Zimbra mail servers make sure you fix their memcache too, it's easy to forget.
I thought it was standard practice to only allow specific IPs access to memcached, either localhost or specific machines in your array - turns out I was wrong.
@Aidan, I tend to agree redis and memcache normally bind to ::1/127.0.0.1 and that's it. I think a fair bit of the problem comes from pre-packaged software that ships with memcached bound to 0.0.0.0.
standard practice would be to have a firewall in place in the first place !
Github was on receiving end at 1.35Tbps sized DDOS attack https://githubengineering.com/ddos-incident-report/
Hetzner also has been suffering loads of packetloss on some of their core routers for the past few days. Support says it's memcached-related.
Remember when I called Hetzner trash? Yeah, it still is.
buyvm also affected by this ddos
We had some compromises but the flood on a shared node this morning wasn't this exact flood
So far I've had to help a half dozen or so clients cleanup their setups, otherwise they'll sit and rim out a full 1Gbit/sec sustained.
Francisco
Has anyone got a clean PCAP of this particular flood? So far I've only seen the info command I wonder if there are others being used.
If you have a large ASN and want it scanned for memcache instances let me know. On a Shodan safari.
ah okay, I saw your tweet so I thing it related
Ah, no, just I had already been helping clients patch so I was hoping to get people doing it without me ticketing.
Francisco
If the software by default listening to 0.0.0.0 then we'll have problem. Just like mongodb fucked by listening on 0.0.0.0 instead of 127.0.0.1. People lazy now days, many sysadmin wanna be just following tutorial on random blog, and left the server as is after they script running fine
memcached usually defaults to localhost/127.0.0.1 don't think i have ever seen memcached use 0.0.0.0 by default
https://news.ycombinator.com/item?id=16493775
wow.. who in the right mind would set it up out of box binding to 0.0.0.0 ! wonder which Linux distros they're talking about ?
A very good amount and mix. From a report on an ISP I just scoped out, 1.4.25 seems to be the most prolific version, not sure what version ships with that currently, I know CentOS 6 and 7 did as well as Ubuntu at one point.
Had a box at RamNode suspended for a short while yesterday supposedly because of this. The box didn't have memcached installed and was properly firewalled. Box was incorrectly identified by automated tools, I suspect.
Please providers, check twice before suspending boxes.
And this is why you need to setup a firewall on every box because of shitty defaults from prepackaged software. SMTP, HTTP, HTTPS, SSH. Everything else should be closed by default.
It's all my fault. That damn UDP port got bound to 0.0.0.0 by default, that's me.
Looks like there is currently Cogent issues (transit level) due to the attacks being thrown around.
Telia and GTT (who we use in our Anycast network) both look fine from my observations in US (L.A, Dallas, Ashburn, Miami) & EU (NL & UK). HE & NTT also tests fine for the test IPs I have west coast.
I don't agree because it is just a workaround for lazy sysadmins who should really check what is listening on a server and why that is and take appropriate action if needed. In other words, sysadmins who know what they are doing. A firewall will not help because the same lazy 'sysadmin' might configure it wrong, not solving anything at all.
IMHO a firewall is only useful if you really need to open a port (for example MySQL and let's say Memcached, lol) and need (not want) to limit is to a few IP addresses / an IP range.
By default it is always your fault, lol.
yep had a few compromised customers hit by this too, was not fun.
A perfectly competent sysadmin might have decided to use memcached with UDP for performance reasons but been totally oblivious RE: it being useful for amp attacks. It wasn't common knowledge until a few days ago.
We've limited it across the board to prevent any significant contributions to attacks and suggest other folks do too
Enabling UDP for performance: okay, fine, but why leave the port open to the world? That is simply asking for trouble like this. Either close it or limit access to a few IPs / an IP range. This way you won't be victim of the next (UDP) port DDOS hype. Beter prevent than correct. This also takes less time in the end and there will be less damage.
A perfectly competent sysadmin might have decided to use memcached with UDP for performance reasons but been totally oblivious RE: it being useful for amp attacks. It wasn't common knowledge until a few days ago.
https://stackoverflow.com/questions/16177084/memcached-authenticating-remote-connections
http://dustin.sallings.org/2010/08/08/memcached-security.html
Honestly still shocked about this... Over here just about every competent sysadmin would've blocked outside access years ago, it's always been bad practice to keep memcache(d) ports open to the world - though it seems to be ignored in many regions.
I was just responding to the part about using it over UDP - I'd definitely say for a configuration involving memcached is a good candidate for being selective in interfaces & firewall - but; listening on UDP doesn't necessarily mean incompetent sysadmin.
Am I missing something here? You were arguing against firewalling it and now you're saying they should?
I said also "IMHO a firewall is only useful if you really need to open a port (for example MySQL and let's say Memcached, lol) and need (not want) to limit is to a few IP addresses / an IP range."
Another person who believes he never makes mistakes and assumes any mistake is caused by incompetence.
Oddly enough, in the real world, I never meet the people who never mistakes. Just on the internet.