Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Linode Suffering Attacks Lately
New on LowEndTalk? Please Register and read our Community Rules.

Linode Suffering Attacks Lately

n1kkon1kko Member
edited December 2015 in General

I've been with Linode for around a year now with no issues but just lately seems they are having a few issues with Dos attacks on quite a few of their locations.

Thanked by 1rokok
«1

Comments

  • cool story

    I ❤ Laravel

  • Yeah, they seem to be getting hammered over the last few days our wordpress blog hosted with wpengine in London is up and down like a yo yo.

  • That's called life.
    They were have success because nobody threats them in past.
    Shame, as for many providers here. What a shame, so big operator and died under ddos attacks. People will never learn basic things: security matters. The good side of the news about linode -> i'm sure, they will have ddos protection in 2016.

  • @desperand said:
    The good side of the news about linode -> i'm sure, they will have ddos protection in 2016.

    So Linode has been running all these years without any DDOS protection? That sounds pretty incredible considering they are a premium provider and many budget providers here have decent DDOS protections.

  • They used to have some form of nullrouting, i don't know whether it was automatic or manual. LET was hosted at Linode in the past and was getting nullrouted for 24 hours when DDoSed.

    -

  • Incredible since they just bought a new HQ and paid $5 millions for it. They should've invested some of that money in better security.

    Thanked by 1xavier
  • @rds100 said:
    They used to have some form of nullrouting, i don't know whether it was automatic or manual. LET was hosted at Linode in the past and was getting nullrouted for 24 hours when DDoSed.

    Clients IP ranges being DDoSed like what was the reason to leave offers for most providers here.

    Always trying to be negative ^^.

  • The one time I was with linode I experienced 3 node failures in a month and they weren't quick solving the issues, always took them more than 5 hours.

    If you consider a provider premium because they accurately inform you about their downtimes then you can find other providers who are a lot cheaper.

    Thanked by 2desperand Rolter
  • Refugee offers ? /s

    I repeat, RAID is not backup | Looking for a developer for your next project? - Hire me

  • raindog308raindog308 Administrator, Moderator

    Linode, KnownHost, WiredTree...none offer DDOS protection even as an addon.

    For LET support, please visit the support desk.

  • In 2015 (soon to be 2016) they should have the infrastructure in place to handle these attacks --- if they dont, it is time to move on.

  • None of them sound exceptionally quality to justify their prices.

    raindog308 said: Linode, KnownHost, WiredTree...none offer DDOS protection even as an addon.

    tr1cky said: The one time I was with linode I experienced 3 node failures in a month and they weren't quick solving the issues, always took them more than 5 hours.

    If you consider a provider premium because they accurately inform you about their downtimes then you can find other providers who are a lot cheaper.

    Go give Vultr(referral) a try. | GNU/Linux http://debian.org

  • @PremiumN said:
    cool story

    Thanks

    Thanked by 1netomx
  • Price of DDoS equipment has dropped a lot

    Different.

  • Each and every Linode host server is now connected via 40 Gbps of redundant connectivity into our core network, which itself now has an aggregate bandwidth of 160 Gbps. Linodes themselves can receive up to 40 Gbps of inbound bandwidth, and our plans now go up to 10 Gbps outbound bandwidth.

    haha

    Thanked by 1inthecloudblog
  • raindog308raindog308 Administrator, Moderator

    GM2015 said: None of them sound exceptionally quality to justify their prices.

    I've used all three. Linode was meh though their panel had some unique features in its day.

    WT support was AWESOME. And out of the box, their servers are very secure - they really go the extra mile to provision cPanel with lots of security enhancements, custom mod_security rules, etc. I really liked WT and if I was a semi-technical user who wanted hand-holding support and easy-to-use managed cPanel, I would use them.

    KH support was almost as good and out of the box, they put in some good stuff, though not like WT. With KH you get more RAM per plan, while with WT you get more disk.

    But none of them have DDOS protection, which just boggles my mind. They could be offering it as a premium add-on.

    Thanked by 1GM2015

    For LET support, please visit the support desk.

  • @raindog308 said:
    Linode, KnownHost, WiredTree...none offer DDOS protection even as an addon.

    They got popular by not being total garbage and advertising heavily on WHT getting recommended by users for months.

    End of line.

  • @Tion said:
    They got popular by not being total garbage and advertising heavily on WHT getting recommended by users for months.

    Not that you are generally wrong, but I received very good support and service from both WiredTree and KnownHost in the past. They were the right providers at the right time for me. Now, that I can manage servers myself, they seem superfluous, but looking back, both did a good job.

    Thanked by 1inthecloudblog

    For those who care:
    You can now find me at https://talk.lowendspirit.com or https://www.hostballs.com

  • I was hesitating to open a thread about this hehe...

    For the capacity of the links Linode claims to have, these should be really big attacks. Unfortunately the servers at my work use the Dallas region, which has been the most affected (until today, which seems to be London).

    This time doesn't seem like a client getting ddos'd, but some haters attacking the whole linode infraestructure.

  • n1kkon1kko Member
    edited December 2015

    @yomero said:
    I was hesitating to open a thread about this hehe...

    I was a little hesitant at first but seems as the attacks are so heavy I thought never mind. They will surely have to bring in DDOS protection now. If not I'll have to move servers.

  • n1kko said: If not I'll have to move servers.

    I was considering to move some important projects, at least to another datacenter, but unfortunately you can't carry your IP addresses across DCs as far as I know. This is important for my particular use cases =(

  • WilliamWilliam Member, Provider

    TinyTunnel_Tom said: Price of DDoS equipment has dropped a lot

    Price of anti-ddos hardware is 80% software and 20% ASICs, SW pricing is the same as usual but 10/40G ASICs got cheaper last year(s).

    spammy said: So Linode has been running all these years without any DDOS protection?

    Yes, only null and enough upstream capacity.

    bogdang said: They should've invested some of that money in better security.

    In some linode locations 100Gbit BW - which is already low on protection these days - costs a million PER MONTH, then you also need the HW on top to actually use that...

  • @William said:
    In some linode locations 100Gbit BW - which is already low on protection these days - costs a million PER MONTH, then you also need the HW on top to actually use that...

    Well, that's interesting how much OVH invested to their anti-ddos protection.

    Always trying to be negative ^^.

  • alexvolk said: Well, that's interesting how much OVH invested to their anti-ddos protection.

    Arbor + Tilera + custom in-house : https://www.ovh.com/us/anti-ddos/

    Passes through a few filter layers, plus all of the BW they have to absorb around their PoPs.

    Thanked by 1alexvolk
  • linuxthefishlinuxthefish Member
    edited December 2015

    @alexvolk said:
    Well, that's interesting how much OVH invested to their anti-ddos protection.

    https://www.ovh.com/us/news/articles/a1171.protection-anti-ddos-service-standard says it costs them around $10 million using traditional filtering, and $103,000 for 100gbit using their own technology.

    If only Linode invested that $5 million they spent buying a building for only 160 people just because the CEO liked it back into their own platform...

  • WilliamWilliam Member, Provider
    edited December 2015

    alexvolk said: Well, that's interesting how much OVH invested to their anti-ddos protection.

    BW in France/CA is 10x cheaper than in JP.

    linuxthefish said: https://www.ovh.com/us/news/articles/a1171.protection-anti-ddos-service-standard says it costs them around $10 million using traditional filtering, and $103,000 for 100gbit using their own technology.

    If only Linode invested that $5 million they spent buying a building for only 160 people just because the CEO liked it back into their own platform...

    This figure does not include any BW.

    I build you 40G filtering at 25k as well, and i need 5k alone for the 40G network cards.

    Thanked by 3Kris alexvolk tommy
  • Which operator/site keep online after targeted by ddoser?
    Come on, you pay ~ $10-$20/m including ddos protection, if you compare to ovh, they will beat all provider in pricing + ddos protection. Why none complaint about delimiter, qps, datashack ddos protection?

    Let's bet which dot-name will collapse first ;)

  • yomeroyomero Member
    edited December 2015

    http://status.linode.com/incidents/mmdbljlglnfd

    "Additionally, we will be announcing the details of an ongoing project to significantly improve our internet connectivity and resiliency. "

    Thanked by 1n1kko
  • FranciscoFrancisco Top Provider

    I don't see them rolling out filtering, I mean, they don't even have their own routers in each of their locations. If you lookup their IP space you'll see that it's being announced by their upstream datacenters and or they peer over a private ASN. If they're having this much of an issue dealing with floods against their platform now, do you think they'll have the time & manpower to deal with leaks on filtering that will happen?

    When LE* was getting hammered back in the day the floods were around ~500mbit. Linode still nullrouted over it, though. The attacker came after us at one point but it never got over a gbit sustained.

    I'm not sure why they rolled out 40gbit uplinks on all of their servers when they obviously can't support it at the core. Sure, you can get some nice (< 2gbit) download speeds on the VM's, but it sounds like they only have a couple 10gigs at the core.

    We'll see, maybe I'm completely wrong and they're eating some cloudflare sized attacks, but I don't see softlayer/coloat sitting on 300~400gbit of capacity in those locations to get knocked offline.

    Francisco

    Thanked by 2Kris vimalware
    BuyVM - Free DirectAdmin, Softaculous, & Blesta! / Anycast Support! / Windows 2008, 2012, & 2016! / Unmetered Bandwidth!
    BuyShared - Shared & Reseller Hosting / cPanel + Softaculous + CloudLinux / Pure SSD! / Free Dedicated IP Address
  • After 189 days, my Newark node kept going on, and NAC is brilliant (8001) - I've colo'd there myself, and they have plenty of bandwidth.

    I honestly don't know what's going on, it's targeted as you saw on WHT Dallas was down right before Newark went down. Some pissed off former employee or pathetic local competitor would be my guess.

    Had to move things to OVH after 1/2 year perfect. From dealing with NAC myself over the years with colo space & transit, 8001 is where you want to be for unparalleled transit, peering, and stability for the east coast. NAC started as a dial up in the 90's, so they have multiple PoPs, LINX peering, etc.

    Not sure where the ball is dropped between robust NAC network -> Linode, but they don't have their own ASN, using 8001, and the prefixes were gone from the route-tables.

    That's when I decided to move. Not even bothering trying to mitigate in RT, but just dropped from the net, all subnets were null-routed.

  • Colocrossing buy a new site secretively? :)

    https://forum.linode.com/viewtopic.php?t=8530

    • in 2012.

    This space for sale, $30 a month.

    1. Kazila - The Worst Xen VPS Hosting Company
  • perennateperennate Member, Provider
    edited December 2015

    http://status.linode.com/incidents/mmdbljlglnfd

    Edit: ah didn't see it was already posted

    I’d like to share some updates about the recent DDoS attacks.

    I am one of several network engineers at Linode who have been working around the clock on DDoS mitigation. While things are stable, I would like to take a moment to publicly address the large and frequent DDoS attacks that we have been receiving since Christmas Day.

    It has become evident in the past two days that a bad actor is purchasing large amounts of botnet capacity in an attempt to significantly damage Linode’s business. The following is a partial list of attacks we have received in no particular order:

    • Multiple volumetric attacks simultaneously directed toward all of our authoritative nameservers, causing DNS hosting outages
    • Multiple volumetric attacks simultaneously directed toward all of our public-facing websites, causing Linode Manager outages
    • Layer 7 (“400 bad request”) attacks toward our web and application servers, causing Linode Manager outages
    • Large volumetric attacks toward our colocation provider’s upstream interconnection points, overwhelming the router control planes and causing significant congestion/packet loss
    • Large volumetric attacks toward Linode network infrastructure, overwhelming the router control planes and causing significant congestion/packet loss

    All of these attacks have occurred multiple times. Over the course of the last week, we have seen over 30 attacks of significant duration and impact. As we have found ways to mitigate these attacks, the vectors used inevitably change.

    As of this afternoon, we have mostly hardened ourselves against the above attack vectors, but we expect more to come. We are working extremely closely with all of our technical partners, including our network equipment vendors and our colocation providers, to prevent future attacks.

    Once these attacks stop, we plan to share a complete technical explanation about what has been happening. Additionally, we will be announcing the details of an ongoing project to significantly improve our internet connectivity and resiliency.

    We would like to apologize for the lack of detail in some of our recent status-page updates. Please know that we are dedicating all resources from multiple departments to stopping these attacks. We acknowledge the amount of downtime we’ve been experiencing is completely unacceptable, and we appreciate the understanding and support we have received over the past several days. We will share more information as our investigation continues.

    Alex Forster

    Network Engineer at Linode

  • I hope a lot of providers start to think about current problems in new year with DDoS attacks, and how mitigate them...

  • Alex Forster should recognize:

    Thanked by 1lifehome

    This space for sale, $30 a month.

    1. Kazila - The Worst Xen VPS Hosting Company
  • xavier said: Alex Forster should recognize:

    So... what about this??? ¬_¬

  • @yomero said:
    So... what about this??? ¬_¬

    Well he retweeted it recently :)

    Thanked by 1yomero

    This space for sale, $30 a month.

    1. Kazila - The Worst Xen VPS Hosting Company
  • @raindog308 said:
    Linode, KnownHost, WiredTree...none offer DDOS protection even as an addon.

    For those of you looking for DDOS protection against your site, you can simply use a service like CloudFlare as a proxy. They have robust DDOS protection, and you're even assigned a CloudFlare IP that masks your real IP. I've yet to suffer from an attack since doing so. There's probably other such companies, but I've never looked into them.

    The current issues at Linode are targeted directly at their upstream provider which is outside of your/their control. Also, if the physical server, in which your virtual server is hosted on, is under attack, there's nothing you can do about that either, which is why they probably don't offer a subscription based option for individual protection. It's up to them to fix it at the physical server level.

    Today's outage has lasted for 9 hours so far!

  • jarjar Provider

    Hats off to their team. I doubt they're sitting over there right now refusing to spend money on it, whatever is going on is spread enough that there's no one quick way out for them obviously. I hate that they had to spend their holiday dealing with this.

    Thanked by 1MikePT

    Founder @ MXroute

  • Pathetic competitor I'm sure. I had to move my machine that was up 189+ days on 8001 (NAC MMU)

    Pissed some little shit(s) are spending the holiday on this.

  • MikePTMikePT Member, Provider

    @jarland said:
    Hats off to their team. I doubt they're sitting over there right now refusing to spend money on it, whatever is going on is spread enough that there's no one quick way out for them obviously. I hate that they had to spend their holiday dealing with this.

    Same. Perhaps they'll invest in DDoS mitigation hardware though.

  • linode deadpooled?

    You're so tight, baby.

  • @creep said:
    linode deadpooled?

    no, they are very old company with agressive adversting, and a lot of clients.

  • Linode has to get a calendar. Not first time they had DDOS outages. Been many years of problems. Good company, but DDOS protect is necessary or some means of dealing with this gracefully (automated nulls). Leadership issue and lack of invest is going to erode their customers.

    This space for sale, $30 a month.

    1. Kazila - The Worst Xen VPS Hosting Company
  • HTTPS slowly became Internet standard, why DDOS protection can't? I hope we get DDOS protection as a standard, not premium features on all webhosting.

    CEO of PT. Rokok Kopi Internet Tidur Tbk.

  • Our Ecommerce website with Linode had like 1 hour of downtime. I mentioned to move the website to the pony.
    Bad thing is that I don't see a 2 gb option with ssd and add a plus for ssd . Also we eat less than 150GB/mo while pony's come with 10 TB which would be an overkill.

    I do like Francisco's products BTW

  • FranciscoFrancisco Top Provider

    @inthecloudblog said:
    Our Ecommerce website with Linode had like 1 hour of downtime. I mentioned to move the website to the pony.
    Bad thing is that I don't see a 2 gb option with ssd and add a plus for ssd . Also we eat less than 150GB/mo while pony's come with 10 TB which would be an overkill.

    I do like Francisco's products BTW

    It's on the high volume table under the main ones :)

    Francisco

    Thanked by 1inthecloudblog
    BuyVM - Free DirectAdmin, Softaculous, & Blesta! / Anycast Support! / Windows 2008, 2012, & 2016! / Unmetered Bandwidth!
    BuyShared - Shared & Reseller Hosting / cPanel + Softaculous + CloudLinux / Pure SSD! / Free Dedicated IP Address
  • NyrNyr Member

    rokok said: HTTPS slowly became Internet standard, why DDOS protection can't?

    Because it's expensive, specially for small ISPs and regions where bandwidth is not cheap.

  • FlamesRunnerFlamesRunner Member
    edited January 2016

    CloudFlare is clearly doing a good job when I try to help someone on LET with a PHP code snippet and get blocked for it :p

    CF Web Application Firewall = complete garbage

    Thanked by 1GM2015

    wget https://s.flamz.pw/dl/bench.sh && bash bench.sh

    curl https://s.flamz.pw/analytics/bench/stats.php

  • ssdvssdv Member, Provider

    Linode run its own network or they are mostly rely on softlayer network?

    turbovps.com
    Offshore Windows and Linux VPS. Sweden/USA

  • yomeroyomero Member
    edited January 2016

    cstreater said: CloudFlare

    No.

    These attacks aren't directed to some site of us, but to someone else or directly to Linode.

    Having cloudflare in our site doesn't help at all.

Sign In or Register to comment.