Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Server suspended due to excessive conntrack sessions
New on LowEndTalk? Please Register and read our Community Rules.

Server suspended due to excessive conntrack sessions

dimitrispdimitrisp Member
edited April 2017 in Help

Hello fellow LETers,

So, I had my server suspended because "it had too many conntrack sessions by our log reports."
I had no idea what conntrack is, until this happened. My question goes mainly to people who have knowledge of this: What can I do to prevent this from happening again?
I've set up a few iptables rules, and I'm thinking of baning most IPs I know I won't use (such as High-risk countries etc), but I don't know how effective it'll be.

Edit: The server is a Nextcloud instance which handles backups of my personal files, and for 3-4 other people as well (all of whom use the NextCloud desktop client). It is also monitored by UptimeRobot and SeFlow's WebSite Protect at 5 minute intervals both.

In need of stable services? Go for BuyVM/BuyShared, PieLayer or SubnetLabs (and also, help me pay some bills)

Comments

  • vmhausvmhaus Provider, Top Provider

    What services do you run on your server? Certain providers do limit conntrack per vps if you were to read their TOS in detailed.

    Thanked by 1dimitrisp

    VMHaus - High Performance SSD SSD Cloud Servers

  • rm_rm_ Member

    This is OpenVZ, right? In which case switch to KVM, most sane KVM providers will not care about your number of connections (unlike OpenVZ where they exhaust a shared resource on the node).

  • vmhausvmhaus Provider, Top Provider

    @rm_ said:
    This is OpenVZ, right? In which case switch to KVM, most sane KVM providers will not care about your number of connections (unlike OpenVZ where they exhaust a shared resource on the node).

    True enough, but even some large providers such as Virmach do have conntrack limit stated on their AUP.

    VMHaus - High Performance SSD SSD Cloud Servers

  • @vmhaus said:
    What services do you run on your server? Certain providers do limit conntrack per vps if you were to read their TOS in detailed.

    Publicly facing: Apache & SSH

    Non-public: MySQL, vnstat, fail2ban and the default of Debian 8.7.

    @rm_ said:
    This is OpenVZ, right? In which case switch to KVM, most sane KVM providers will not care about your number of connections (unlike OpenVZ where they exhaust a shared resource on the node).

    Yes, it's OpenVZ. At the price, I doubt I can find a KVM machine with such specs.



    The provider sent me a snip of the conntrack logs, and I can see it's an IP that tried to connect to my server, that I don't recognise. Perhaps someone tried to DDoS my IP or hack into my server (although fail2ban should have kicked in)

    In need of stable services? Go for BuyVM/BuyShared, PieLayer or SubnetLabs (and also, help me pay some bills)

  • rm_rm_ Member
    edited April 2017

    dimitrisp said: such specs

    Something ridiculous like 6 GB RAM? That's likely oversold by 10x, you will find much better performance and stability at any honest 2GB KVM (such as the OVH VPS-SSD for 3 EUR) compared to that.

    Or do you mean a super low price like 12 USD/year or so.

    dimitrisp said: The provider sent me a snip of the conntrack logs, and I can see it's an IP that tried to connect to my server, that I don't recognise.

    A lousy provider to begin with, suspending a VPS not for outgoing flood, but for (unwanted) incoming connections. Probably don't even have a DDoS protection too.

  • It's not something ridiculous like that. It's 200GB HDD, 2TB Bandwidth, 1GB RAM & 1IPv4 for €20/year.

    I may be wrong about the incoming/outcoming connections log file, as I am at my office job and can't properly analyse it. As for DDoS protection, well, it's not needed as it isn't used for anything other than Nextcloud.

    [I'm trying to keep a balance on my posts, as I'm not sure who's to blame 100% yet: it might be my fault because I overlooked something, it might be something faulty on the provider's end, and I don't want to come across as a "needy, mean & ungrateful LET kiddy". But in the 8 years I'm using VPSes, it's the first time I got suspended, and it is somewhat alarming. The provider unsuspended the VPS in the same day, after I send a ticket]

    In need of stable services? Go for BuyVM/BuyShared, PieLayer or SubnetLabs (and also, help me pay some bills)

  • Probably nothing faulty on the providers end, but they could suspend for a small number of conntrack sessions which could be reached with legitimate usage.

    Maybe @cociu can offer you similar specs for 20€/y, based on his past offers.

  • What was the number quoted out of interest?

    KVM or not at some point it is going to overwhelm the bridge.

    I am no longer active here, find me at https://talk.lowendspirit.com

  • @AnthonySmith said:
    What was the number quoted out of interest?

    KVM or not at some point it is going to overwhelm the bridge.

    VPS (redacted) has 30290 conntrack sessions.

    In need of stable services? Go for BuyVM/BuyShared, PieLayer or SubnetLabs (and also, help me pay some bills)

  • BopieBopie Member

    If its used for only personal stuff I see no reason for those many connections, My advice which may not be the best so don't rely on the advice, lockdown the server, Change ssh port and limit connections to only your ip i.e block all connections except your own IP maybe this well help just in case it is a bot trying to brute force or an attack

    Thanked by 1dimitrisp

    Aaran.cloud
    When I was a child, I spoke as a child, I understood as a child, I thought as a child, but when I became a man, I put away childish things

  • @dimitrisp said:
    It's not something ridiculous like that. It's 200GB HDD, 2TB Bandwidth, 1GB RAM & 1IPv4 for €20/year.

    That's btw not unreasonably far away from KVM prices. The problem might be the hdd size and, to a lesser degree, the bandwidth. How much of that do you really use and need?

    Thanked by 1dimitrisp

    My favourite prime number is 42. - \forall cpu in {intel, amd, arm}: cpu->speed -= cpu->speed/100 x irandom(15, 30) | state := hacked

  • dimitrispdimitrisp Member
    edited April 2017

    @Bopie said:
    If its used for only personal stuff I see no reason for those many connections, My advice which may not be the best so don't rely on the advice, lockdown the server, Change ssh port and limit connections to only your ip i.e block all connections except your own IP maybe this well help just in case it is a bot trying to brute force or an attack

    SSH port was changed on day 1 of VPS provisioning. It's like the first or second thing I do on a new VPS.

    As for limiting connections to my IP, that is impossible. Dynamic IP on home, and ISP has multiple subnets, as well as I'm on 3 or more different ISPs throughout the day. Also, 3-4 people that are using the same server, are on different Dynamic IP & different ISPs as well. Just thinking about it is a headache :P

    I could though whitelist all my country's IPs, but I don't know how viable that would be, as I might occasionaly share a file with a friend on the other side of the globe (didn't happen in the past 4 months though)

    @bsdguy said:

    That's btw not unreasonably far away from KVM prices. The problem might be the hdd size and, to a lesser degree, the bandwidth. How much of that do you really use and need?

    I am currently using ~40GB out of 200GB, and with the upcoming personal projects I have planned, I expect it to reach at least 100GB by the end of the year (but I can't talk about the others using the server, I don't know what they have on their minds). As for the bandwidth, since I am syncing, and not downloading every time I need something, my usage is very low as you can see on the image below:

    In Nov'16 I transfered all data to this server from my old one, and this month I had a catastrophic HDD failure at my work computer.

    In need of stable services? Go for BuyVM/BuyShared, PieLayer or SubnetLabs (and also, help me pay some bills)

  • bsdguybsdguy Member
    edited April 2017

    @dimitrisp said:

    >

    I am currently using ~40GB out of 200GB, and with the upcoming personal projects I have planned, I expect it to reach at least 100GB by the end of the year (but I can't talk about the others using the server, I don't know what they have on their minds). As for the bandwidth, since I am syncing, and not downloading every time I need something, my usage is very low as you can see on the image below:

    (image removed)

    In Nov'16 I transfered all data to this server from my old one, and this month I had a catastrophic HDD failure at my work computer.

    Hmmm, 200 MB hdd might bite you with most providers. But if you can live with 512MB RAM and 1TB or even 500GB bandwidth (which should be easily feasible for you) you might find a provider with a KVM in the 20$ to 25$ range.

    My first address to go to and ask would be @cociu. He is flexible enough and crazy anyway (crazy in a good sense).

    My favourite prime number is 42. - \forall cpu in {intel, amd, arm}: cpu->speed -= cpu->speed/100 x irandom(15, 30) | state := hacked

  • It's worth asking the provider if there is a hard limit. It might be too low for what you need, regardless of what you try to change.

  • get a DDoS protected server then .

  • Yeah I can see why that might be an issue haha.

    dimitrisp said: VPS (redacted) has 30290 conntrack sessions.

    I am no longer active here, find me at https://talk.lowendspirit.com

  • raindog308raindog308 Administrator

    I'm reaching into my back memory here but I think conntrack -L or -l will list all the connections. If it's not that command then you can find it via google. You might be interested to see where all those connections you don't know about are from. Perhaps some firewall rules would clean things up -?

    Usually I only run into this when doing p2p which has a ton of connections. I agree with the comments that you might be happier on KVM.

    For LET support, please visit the support desk.

    LowEndTalk attracts the finest members. - bear, WebHostingTalk

  • BopieBopie Member

    @AnthonySmith said:
    Yeah I can see why that might be an issue haha.

    dimitrisp said: VPS (redacted) has 30290 conntrack sessions.

    The last time I saw that many conntrack sessions was on a VPS hosting a Chinese search engine XD

    Aaran.cloud
    When I was a child, I spoke as a child, I understood as a child, I thought as a child, but when I became a man, I put away childish things

  • @AnthonySmith said:
    Yeah I can see why that might be an issue haha.

    Figured so! After a little bit of research, I saw that most providers have a cut off around 30k sessions, so I'm glad I didn't get terminated! Haha

    @raindog308 said:
    I'm reaching into my back memory here but I think conntrack -L or -l will list all the connections. If it's not that command then you can find it via google. You might be interested to see where all those connections you don't know about are from. Perhaps some firewall rules would clean things up -?

    Usually I only run into this when doing p2p which has a ton of connections. I agree with the comments that you might be happier on KVM.

    I run this: cat /proc/net/nf_conntrack
    Right now I see around 20 connections, which is reasonable I think. I recognise most of the IPs, and I'm in the process of searching and baning the unknown ones.

    To be honest, there is no P2P software running. I'll see if I can do something with IPTables and banning whole countries (If someone can help with that, feel free to point me in the right direction)
    I'm a reasonable person, if this reoccurs and the provider asks me to leave, I'll do so, but it'll be a pitty since I'm 95% positive I didn't do something wrong. So far everything checks out, so it might have been a randomly-targeted attack. I'm saying 95% as there's nothing running/installed that I don't know what it is, the SSH log shows no denied attempts from strange IPs etc.

    In need of stable services? Go for BuyVM/BuyShared, PieLayer or SubnetLabs (and also, help me pay some bills)

  • cubedatacubedata Member, Provider

    @dimitrisp said:

    that sounds like nodewatch is running on the nodes.
    you might want to check your vps and see what is on it as it could be; cannot be sure of this but the provider has the nodewatch connection track limit set to low?
    maybe discuss this with your provider?

    Check out all of our custom modules here: https://cubedata.net for both blesta & whmcs.

  • yeah random attacks happen, openvz is obviously more sensitive in this regard as it is a single kernel essentially, one of the drawbacks but also one of the reasons the price is low.

    While it is true that KVM is better equipped and I understand why end users that have never really done any serious work virtualization think it is the answer to everything, the reality is that in most cases it is the host node managing the bridge and all the traffic has to pass through that first, so yeah KVM helps, it is not open license to run 65k 24x7

    OVH is cheap, but they are also pretty crap when it comes to monitoring (or caring) what is going on, they run on scale with no real concern for quality, that is why they have the reputation they have for people running seedboxes etc, expect bad neighbors if you go that route.

    Thanked by 1dimitrisp

    I am no longer active here, find me at https://talk.lowendspirit.com

Sign In or Register to comment.