Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Providers, how common is node abuse by consumers today and what do you do to combat it?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Providers, how common is node abuse by consumers today and what do you do to combat it?

pubcrawlerpubcrawler Banned
edited December 2012 in General

To the folks that are VPS providers:

How common is node abuse by your customers today?

What do you do to combat it?

«134

Comments

  • It's weekly, we deal with it by using monitoring systems and abuse scripts. As well as manual inspections.

  • BTW: This thread is from another busy one now about node abuse.

    I am not a provider, but, think some of the newer providers and those who haven't dealt with abuse issues could benefit from the collective knowledge of those who have been there and done that.

  • Deal with it daily

    Most comes from inexperience of the user like poor scripts, Deluge/torrents, and OCD idiots running benchmarks all day

  • What counts as "abuse"? I've read some providers you can't use >% amount of CPU, RAM or you use too much IO etc.

    Theres also the piracy excuse that I've had experience with using a VPS provider on here.

    At the end of the day, it seems simple to me.

    Piracy = DMCA notice, abuse.
    Spam = IP Blocklist/Complaint, abuse.
    CPU = 99% Usage 24/7, abuse
    RAM = Would using burstable ram for a long period of time be considered abuse? I'm not too sure how that works tbh.
    IO = If you are paying for a 100Mbit connection you should be able to use at least that for IO.

  • Not very often, I don't have a abusive clientbase, the occasional SPAM Report but no major resource abuse issues.

  • Oh, don't offer Windows VPS servers - your abuse/hand holding clients will disappear overnight

  • Just to add to this discussion, I posted a similar thread a while ago:

    http://www.lowendtalk.com/discussion/6465/how-do-you-guys-handle-clients-who-abuse-io-other-resources#Item_27

    Some useful replies there

  • jarjar Patron Provider, Top Host, Veteran
    edited December 2012

    @spycrab101 said: Piracy = DMCA notice, abuse.

    Spam = IP Blocklist/Complaint, abuse.
    CPU = 99% Usage 24/7, abuse
    RAM = Would using burstable ram for a long period of time be considered abuse? I'm not too sure how that works tbh.
    IO = If you are paying for a 100Mbit connection you should be able to use at least that for IO.

    Good points. Here's how I'd outline my responses as a provider.

    Piracy - You get a DMCA, I ask you to stop. Another, terminated.
    Spam - Terminated. Perhaps if you've been a client for a while I'll start a conversation first to see if your VPS was compromised. If you sign up today and I'm on black lists in 4 hours, don't expect any courtesy.
    CPU - I'd prefer you didn't cap more than two cores for an extended period of time. I'd prefer you didn't cap more than one core for a very extended period of time. In almost any case I'm going to open a dialogue or, if it's deluge...cpulimit.
    RAM - You should be able to use what you purchase. If tomorrow everyone decides to use 100% of what they purchase, it's my obligation to make sure that they can. That'll never happen though.
    IO - I don't care as long as iowait is low. Once it starts to climb, I'm looking for the offender. If I can solve abuse of it with cpulimit and io priority, that's what I'll do. Most of the time this is fine. Most people don't seem to mind a bit of limitation, they just don't know how to apply it on their end.

  • HaveDroidHaveDroid Member
    edited December 2012

    @bamn said: Deal with it daily

    Most comes from inexperience of the user like poor scripts, Deluge/torrents, and OCD idiots running benchmarks all day

    I haven't run torrents (and deluge would be my preference) on an LEB but it's certainly something that has entered my mind. What is the problem? IO usage? Complaints?

  • @HaveDroid said: What is the problem? IO usage? Complaints?

    All the above

    Excessive connections by a misconfigured torrent client which affect others. If you try to explain to turn it down to something reasonable, a Southeast Asian language barrier is always fun

  • @bamn

    When seeding transmission uses about 5% CPU. I have noticed some other torrents use insane ram like rtorrent. Ram usage with transmission seems to hover around 50mb-70mb regardless of the number of connections.

    I've not been monitoring IO, is there a good application that can do so? So I can post those stats.

  • @spycrab101 said: When seeding transmission uses about 5% CPU. I have noticed some other torrents use insane ram like rtorrent. Ram usage with transmission seems to hover around 50mb-70mb regardless of the number of connections.

    $4-5 "custom" VPS
    1 CPU core
    Barely enough RAM to run XFCE
    Spends all the money on bandwidth

  • @bamn

    I'm using a similar setup and I'm not running into any issues. However rtorrent is a terrible torrent client it hogs everything, hence I'm using transmission.

  • jarjar Patron Provider, Top Host, Veteran

    @HaveDroid said: I haven't run torrents (and deluge would be my preference) on an LEB but it's certainly something that has entered my mind. What is the problem? IO usage? Complaints?

    I find that if you limit deluge to 5% of a CPU core, it has a hard time working hard enough to need the IO that it uses when uncapped... so no complaints here. My clients running deluge without any limitations in their configuration might have a complaint there but meh, they're still getting what they want ;)

  • I see Torrents themselves as abuse, therefore I do run my own dedicated server for them.

  • @HalfEatenPie said: I see Torrents themselves as abuse, therefore I do run my own dedicated server for them

    Isn't that the reason kimsufi 2G boxs exist? :P

  • Insightful.

    @Jarland, again, I love your approaches to things. Very similar to my thoughts/approaches.

    As for torrents, I can understand how they are a nightmare. Out of box defaults tend to have way too many network connections and drives IO into the floor. If scaled back properly, can be fine, but if a provider allows torrents I think they should put a FAQ recommendation piece together on preferred clients and how to limit the torrent flooding traffic and IO smashing.

    Let's see what else:

    Piracy = DMCA notice, abuse----

    Been there. Had a Tor node on a dedicated and DMCA complaints rolled in. So I just discontinued Tor. About that time I took a disliking to Tor and a study came out about rogue end nodes snooping and piecing traffic together, so it became clear it was useless.

    Spam = IP Blocklist/Complaint, abuse.

    I wish providers took this serious. Especially the annoying bot comment spam. I am getting deluged by it right now from one of my least favorite providers Quadranet and some goon on a Miami node of theirs.

    CPU = 99% Usage 24/7, abuse

    Perhaps recommending 'nice' usage to clients is a start for Linux newbies. Again, the FAQ!

    RAM = Would using burstable ram for a long period of time be considered abuse? I'm not too sure how that works tbh.

    RAM use should never be considered abuse as you are allocated your limits and swap. If you hit the limits then your VPS should suffer from it, but should still work.

    IO = If you are paying for a 100Mbit connection you should be able to use at least that for IO.

    Bandwidth is another touchy oversold part. Providers will sell you a Gigabit connected VPS, but hope you use the 1TB 2TB whatever TB in semi linear fashion. If you get popular for 48 hours on Reddit or somewhere else, then blow up happens. A provider should be happy and glad to charge you for the overage when it occurs.

  • spycrab101spycrab101 Member
    edited December 2012

    Here's my usage during torrenting. Downloading and Seeding and how long I normally do these for.

    https://dl.dropbox.com/u/60865381/linuxtransmission.png

    I assume the high CPU usage and the IO are connected? If I limited the IO would I see a reduction in CPU usage? Would a VPS provider consider these high spikes as abuse? I would be interested to know.

    I would like to add that doing this on rtorrent/Deluge would be impossible due to the fact they are no where near as good at managing resources as transmission. I don't understand why people use those clients on a VPS.

  • A good old saying,
    There is no such thing as burst ram ;)

  • @Taz

    I've only been given 128MB of RAM however htop has reported 256MB ever since I've had this VPS. I'm still kinda confused as to how burst RAM works. It it normally okay to use more than 128mb?

  • From the original ovz documentation burst ram should be your true memory limit because of the screwy way ovz does memory management.But providers have changed the term to fit their own marketing need.

  • jarjar Patron Provider, Top Host, Veteran

    @spycrab101 said: Would a VPS provider consider these high spikes as abuse? I would be interested to know.

    If the usage you're showing in those screenshots is accurate and that high write only happens for 1 minute every 6 hours, personally I would let it slide. If you were running downloads like that all day long with that kind of IO, I'd have a fit ;)

  • KuJoeKuJoe Member, Host Rep
    edited December 2012

    @pubcrawler said: How common is node abuse by your customers today?

    Not very common anymore. During our first 6 months we had constant abuse to the point where we would sleep with one eye open and when we received an e-mail we always assumed it was from high loads. Our biggest issue these days is people hogging a single CPU core for hours at a time which isn't fair to other clients (assuming a node has 8 cores and 16 client it's not fair to make 15 clients share 7 cores while 1 client gets 1 all to himself).

    What do you do to combat it?

    We have scripts in place for our OpenVZ node to monitor both node and individual VPS loads (i.e. the output you receive from uptime which is impacted by both CPU and disk IO).

    If a VPS exceeds a certain load level for X minutes (I forget the setting off hand) the VPS is restarted.
    If a client exceeds 99% load (1.00 = 1 CPU core) for over 15 minutes we get an e-mail telling us about it so we can investigate if other VPSs are fighting for CPU or if the client is doing something they shouldn't be (most of the time it's from Folding@HOME type of scripts that are designed to use up all idle CPU which angers me to no end).
    If a node exceeds a certain load level for 1 minute we get an e-mail telling us the top 10 processes, iotop output, and VPS loads.

    Notice all of these scripts are load related. There are ways to restrict CPU but but just adds to the load because more and more things are queuing up for the CPU. We had played around with the idea of giving all VPSs access to all CPUs so that they could finish their processing faster but we then found clients who would hammer all of the cores just because they had access to them (and killing processes over SSH were painful to do with a load of 50+).

    Disk IO is a monster in and of itself because there is no real way to limit it. There have been talks of a disk IO throttle kernel module but I've never seen any place that ever got it to work and development/support stopped many years back. Luckily disk IO hasn't been an issue since moving to a RAID10 setup (knocks on wood).

    Network abuse is the next biggest thing after load abuse. We run a script to check the LAN and WAN speeds of each node and if it falls below a certain threshold we investigate. Since we started using a non-SolusVM solution to network throttling this hasn't been an issue but we still find some clients running scripts that violate our TOS and burn through bandwidth which can impact other clients during peak times when everybody is running backups or is getting high volumes of traffic. In the past 6 months we've had maybe 4 or 5 clients that decided to run open public proxies and were using more bandwidth in 12 hours than the rest of our clients combined (and also resulted in more abuse reports than the rest combined also). Outbound DOS attacks are also painful but we combat this at the router.

  • So far everyone talks about ovz. What about kvm?

  • KuJoeKuJoe Member, Host Rep

    @Taz said: What about kvm?

    KVM is really good for resource management out of the box. Based on my tests, CPU isn't an issue because it will just divide the load (i.e. if 2 VPSs want to use a single core they each get 50% of the processing). Disk IO isn't an issue because it is also divided up (running a dd test on 2 different VPS will result in each VPS getting exactly 50% of the MB/s as a single VPS running the same dd test).

  • Chweet!

  • oh boy here we go..

    @KuJoe

    If a VPS exceeds a certain load level for X minutes (I forget the setting off hand) the VPS is restarted.

    Why don't you just kill the process that's causing the problem instead? Then email the client telling them why you had to kill the process. Seems simple enough to me.

    If a client exceeds 99% load (1.00 = 1 CPU core) for over 15 minutes we get an e-mail telling us about it so we can investigate if other VPSs are fighting for CPU or if the client is doing something they shouldn't be

    Can't you just set it up to email both you and the client before turning things off? When you cut my VPS off I never even knew about it until 10mins later. Leaving me itching my head figuring out what the hell was going on.

  • KuJoeKuJoe Member, Host Rep
    edited December 2012

    @spycrab101 said: Why don't you just kill the process that's causing the problem instead? Then email the client telling them why you had to kill the process. Seems simple enough to me.

    Because that can be extremely dangerous. We would rather reboot a single VPS than crash an entire VPS node. If we are manually investigating a high load on a node, then we may kill the process manually but for automation we only allow our scripts to issue a 'vzctl restart' command because this is the safest command I can think of to resolve the high loads, allowing any automated script that is running as root to kill processes without any manual checking is a very bad thing and something I will never allow on our servers.

    @spycrab101 said: Can't you just set it up to email both you and the client before turning things off? When you cut my VPS off I never even knew about it until 10mins later. Leaving me itching my head figuring out what the hell was going on.

    We would like to do this and some day when my coding abilities are better I will consider looking into this option since I've never pulled data from a database in a shell script before nor do I have time to do this any time soon (my ToDo list is insanely long and the "if it's not broke, don't fix it" stuff is lowest priority).

  • Because that can be extremely dangerous.

    My openvz knowledge is nearly 0 I must admit so I don't understand how killing a process on a clients vps can take down an entire node? That sounds absolutely ridiculous. Is this an openvz problem?

Sign In or Register to comment.