Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


CPU Abuse - WTF?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

CPU Abuse - WTF?

I've just had a VPS (running nothing but an idle OpenVPN server) suspended by my provider due to "CPU Abuse." I guess that's what happens when you get a bit too cheap?

Is this sort of policy due to limitations with OpenVZ -- can you not limit the CPU used by a VPS in the same way you can limit RAM, etc use?

I guess I'd have expected that OpenVZ/Solus/etc would allow you to cap the CPU used by an individual VPS, or limit each to its "fair share" if multiple VPSes were engaging in heavy CPU usage at once. I've never had to do this on my own hosts but it looks like LXC, which I use, is capable of limiting CPU consumption of individual containers.

Thanked by 1pluush
«1

Comments

  • Have you contacted your service provider and explained what you have running on your VPS, If you believe it is an error then I'm sure they would be willing to help sort the situation with you

  • What's the provider ?

  • Yeah, contacted them over 24 hours ago ... they're clearly much faster at suspending service than responding to tickets. I'm not expecting brilliant service when I'm paying the equivalent of a couple of hours at minimum wage for a year's VPS hosting.

    I'm just amazed that this is even a thing, and wondering if it's something everyone does due to OpenVZ not allowing CPU limit setting, or that feature not working properly, or something else.

  • You can cap CPU on OVZ. This is just a sign of a poor quality host, imo.

  • As said you can potentially cap CPU, however, this is not often the approach of a host, they simply allow you "fair share" of CPU and of course when you abuse that fair share they suspend you, @deadlyllama I do find it terrible that they have not responded to you in 24 hours after suspending you, however like you said if its cheap likely the support will be slow, if its poorly setup OVZ I would imagine that it's simply an error on the providers side and not monitoring correctly but I may be wrong.

  • jvnadrjvnadr Member
    edited March 2017

    It's not working like that. Most of providers do allow you to eat even a whole of a cpu thread for a short period of time, but they ask from you to monitor the server, so be sure this is not happens a lot. This is even more often as policy when it comes to such cheap yearly services.
    They suspended you, you say. I think this is normal and happen a lot.
    If your provider did cap cpu usage, he wouldnt let you use some more power for spikes. Some do cap it heavily and this leads sometime even in difficulties to install a simple software...
    Suspension can be automated, ticket answering is not. And if you pay so little for a year, then, you should wait a bit for a response. Be polite and try to determine with your provider what happened and you vps had reached cpu high. And, if the service is critical (or, better, even if it's not), a monitoring system like observium could show the behavior of a server and send alerts when something is not going well.

    Thanked by 1PioHost
  • I much prefer the way @exception0x876 does limits. Actual fair share. If 4 people on a server all use all the power available, they all get 25%. If only 1 person uses all power available he gets 100%.

    Thanked by 1sayem314
  • deadlyllamadeadlyllama Member
    edited March 2017

    If it's possible to cap CPU, why not just do that when you detect "CPU abuse" rather than totally suspending the VPS? This is roughly what AWS do on their cheap t1/t2 instances -- they let you burst up to full CPU for a while, then throttle you back.

    I guess my surprise is because I'm from a software engineering + sysadmin background. I'd have thought automatic, temporary throttling would cost you way less in staff time than automatic VPS shutdown and a corresponding bunch of tickets and unhappy customer.

    The last time I ran into a manually imposed limit like this was with a dodgy dedicated server provider in the mid 2000s, who would unplug your server if you used more than 4MBps for more than a few hours in a row.

    I got excited by the promise of a local (NZ) VPS at a very low price, a combination that doesn't come up very often. If it gets randomly shut down then it's no more reliable than hosting on the old PC in my garage.

    Thanked by 1pluush
  • MikeAMikeA Member, Patron Provider
    edited March 2017

    Did they mention load anywhere in the ticket to you? CPU on your server can only be 50% but your server might be causing very high load on the host for some reason. This is the joys of OpenVZ and not KVM really. If the suspension is literally just high use of your allocated cores then yeah it's a bit much.

  • DanDan Member

    Shoutout to KgoVPS, they've treated me very well at the LET pricepoint I pay. Plus they worked with me on rollover bandwidth and a few other things, Kro is a great guy.

  • pbgbenpbgben Member, Host Rep

    Harmless banana hijack. Check the sig for some sweet SYD based KVM vps

  • Should have used softether...

  • I wonder if the vpn software went berserk and started eating cpu for some reason. I haven't seen openvpn do that but I've seen it with some other programs including apache.

  • AnthonySmithAnthonySmith Member, Patron Provider

    Hey there is a simple solution, ask your host to limit you to 50% of 1 core on a lower priority, then you can never really abuse it, they will be more than happy to do this because you are the exception.

    Most people want to be able to burst up and use 6,12,24GHz for short periods to get things done and most people would go bat shit crazy if they bought a vps and got half a core.

    Thanked by 2jvnadr netomx
  • Even with a capped cpu you could impact performance of the whole node on openvz.

  • which vpn are you using?

  • NeoonNeoon Community Contributor, Veteran

    Switch the Provider, which does have a CPU Usage Policy or get a Dedi, best thing I can recommend, if they take to long to reply or you do not get a solution from them.

  • VirMachVirMach Member, Patron Provider

    The main benefit of virtualization is lost when you want strict lines drawn. By giving you more resources and having a fair use policy, you get more bang for your buck. Providers could give you dedicated resources or limit you specifically to what you pay for, but I don't understand how that benefits you as a consumer.

    Are you really stating that you would rather have only 50% of a core at all times, rather than have the 50% (at the same cost) but also be able to burst to 100% when required?

    AnthonySmith said: Hey there is a simple solution, ask your host to limit you to 50% of 1 core on a lower priority, then you can never really abuse it, they will be more than happy to do this because you are the exception.

    Exactly. But I suppose people like to see the higher numbers when they're purchasing the service at a low cost, and then compare it to higher cost services that permit you to use all the resources while criticizing the host for not limiting them. Even if they have the option to purchase more resources and stay within the fair usage policy, they would rather argue. For whatever reason, some people would rather pay $10 for a product with 1x resources than pay $10 for a product with 2x resources that you can use half the time then they like comparing it to the $20 product with 2x resources.

    deadlyllama said: I guess my surprise is because I'm from a software engineering + sysadmin background. I'd have thought automatic, temporary throttling would cost you way less in staff time than automatic VPS shutdown and a corresponding bunch of tickets and unhappy customer.

    This solution would definitely work best, if your provider did have a system in place to just throttle your usage automatically if you break their fair use policy. Unfortunately, that's not really how it usually works by default. For CPU on OpenVZ it may be easy to do, but imagine attempting the same scenario for other resources or virtualization. All it would do is create a confusing idea of how the company handles abuse.

    Our solution to this dilemma was giving customers an option to purchase permission to use the other half and bypass our fair use policy, but that doesn't necessarily mean all customers who break fair usage are interested in paying more, lowering their usage to abide by our fair use policy, and it definitely doesn't mean they won't complain. So that doesn't really save any staff time. What does save staff time (and result in a better performing server for all customers) is enforcing fair usage policies when necessary. Of course it would have been more appropriate for your provider to contact you first. Maybe they're a bad company and they are very strict with poor support, or maybe you were causing a high load so they took immediate action.

    MagicalTrain said: I much prefer the way @exception0x876 does limits. Actual fair share. If 4 people on a server all use all the power available, they all get 25%. If only 1 person uses all power available he gets 100%.

    This might create a bad environment where the 4 people all expect 50%, for example, but only get 25% and they are unable to run what they want. It's basically competing for resources and not very suitable if you end up with bad neighbors.

    Imagine the same concept with disk. In most cases it is "actual fair share" but if the provider just lets you duke it out to use all the resources it may create an environment where people who want to be able to use all the I/O they want all end up on the node and completely ruin the server's performance. Again, this also results in customers expecting more and receiving less to an extreme level which will still result in complaints - just from a different crowd. Fair usage policies that allow you to burst to a lower amount than "actual fair share" but that give you more resources than "dedicated" are the perfect middle ground.


    With all that being said, there's different price points for everything and different boundaries. It's up to you as the customer to decide what will work best for you. If you're running something that will ALWAYS use high amounts of resources, go for dedicated resources. If you need to just burst for short periods of time and want to be able to do small things quickly, then go for the "actual fair share." If you're somewhere in between (where most customers end up) go for a service with a fair usage policy that fits your needs.

    But if you selected the wrong type of service for what you're doing and have different expectations than the reality, don't be shocked if it results in a bad experience for everyone involved.

  • raindog308raindog308 Administrator, Veteran
    edited March 2017

    VirMach said: The main benefit of virtualization is lost when you want strict lines drawn.

    I'd say "a benefit". There's a benefit to simply carving up big hardware, even if allocations are fixed, but yeah, there's an inverse relationship between full utilization and strict lines.

  • raindog308raindog308 Administrator, Veteran

    This thread seems relevant:

    https://www.lowendtalk.com/discussion/95633/reasonable-cpu-use-a-look-at-aups

    tl;dr: some providers explicitly call out limits/policies in their AUP, and some don't. In the latter case, how is the user to know what's reasonable?

    @VirMach you were the first listed on the scale going from "most specific" to "most vague" :-)

  • YuraYura Member

    @Virmach, I'm very happily running one service on your black friday windows special. Yet I was more than a little bit worried when system updates had to be run... During normal operation my server doesn't break any limits, cpu runs under 10%, I recall, but maintenance or windows weirdness happens (system update consumes 50% of the server) and I dread to be suspended for surpassing your CPU policy of "not more than average of 50% in 2hrs, or 90% in 15 minutes".

    There is nagging feeling. But so far, nothing bad happened. Good service.

  • ucxoucxo Member

    Does VPSDime still do that thing where they hard-reboot your VPS when its load exceeds 1.0 for more than one hour (or whatever the exact threshold was)?

  • VirMachVirMach Member, Patron Provider
    edited March 2017

    raindog308 said: there's an inverse relationship between full utilization and strict lines.

    raindog308 said: @VirMach you were the first listed on the scale going from "most specific" to "most vague" :-)

    Being the most specific in the AUP doesn't necessarily mean the most strict lines. I would like to think being the "most specific" is beneficial to the customers, as they have better guidelines on what is acceptable. While vague acceptable usage policies might look less scary, it actually usually just means the host can suspend you for whatever they deem to be acceptable at the time for any reason.

    I wouldn't necessarily like to describe it as "usage is frowned upon." There's only so much VMs that go on a single server and in many cases users aren't really using their resources. We're also usually more lenient than the numbers published on our AUP. The numbers are just there to give you an idea of what would be the acceptable usage, if you end up on a node where everyone does want to use all their resources. On most of our full KVM nodes, CPU usage is at around 50% and I/O usage is low as well. So there's definitely resources to go around, but the fair usage policies are there to keep performance optimal at all times.

    Yura said: During normal operation my server doesn't break any limits, cpu runs under 10%, I recall, but maintenance or windows weirdness happens (system update consumes 50% of the server) and I dread to be suspended for surpassing your CPU policy of "not more than average of 50% in 2hrs, or 90% in 15 minutes".

    We have been working on our automated systems for a long time on our KVM services to reduce false positives. They do still occur, but we do listens to customers and investigate further to verify these rare situations.

    As I stated, the numbers in our AUP are generally where we begin even looking toward your service as possibly being abuse. That's when it gets entered into our abuse system, but that does not mean we take any actions. As for our OpenVZ services, most servers are at 10-20% CPU usage and we generally don't care if you use more than the AUP. That's why we recently increased the CPU offered with most of our OpenVZ packages. For OpenVZ, we generally just take a look and if you're not blatantly overloading the server or literally maxing out CPU 24/7 without the high CPU option, you should be fine.

    raindog308 said: I'd say "a benefit". There's a benefit to simply carving up big hardware, even if allocations are fixed

    I said "main benefit" which implies there are obviously other benefits. I definitely would argue that it's the "main" benefit to be able to burst on a VPS. You can't burst on a small dedicated server but it also ends up costing more in most situations. Of course there are benefits going the "virtual dedicated" route but you lose the benefit of low cost and/or are given less resources in the end.

    My argument is that 2 cores (with 50% AUP) is better than 1 Core (truly dedicated) at the same price. In fact, the 2 cores (with 50% AUP) ends up costing less to the provider and consumer because for a provider to give you 1 Core (truly dedicated) it means that the server is more empty, even if no one is using the resources. That's lost efficiency right there which ends up costing everyone more money. A host may be able to provide 20 cores (with 50% AUP) on an 8 core server and most people will use less than the 50% so it would work out @ 1/20th of the price of the server. However, if the same host provides 8 cores (truly dedicated) on an 8 core server and most people use less than 50%, then over half the server's processing power would be wasted and it would be 1/8th of the price of the server. Remember, in both situations (assuming the server in filled properly and not overloading) you are getting the same amount of "power" allotted to you, and on the cheaper server (2.5x cheaper) you can actually burst to double what the more expensive service has allotted you. So yes, I would definitely say that's the main benefit of virtualization. Again though, the "main benefit" only applies to the "main" portion of VPS users - there are obviously people who would benefit from virtual dedicated resources where they're permitted 100% usage, which is why we also offer a "high CPU" option. However, as I stated, there's still customers that want the price of an AUP without an AUP. That's why I don't understand people that specifically complain about our CPU AUP - during checkout you're specifically encouraged to purchase the high CPU addon to subsidize the cost of permitting you 100% usage. We would absolutely love to let customers use what they want and pay for what they use, but in some situations it's difficult to do that. For example, I/O is a completely different story as a couple VMs can trash the entire server's disk. From our developer's understanding, it's difficult to properly throttle I/O and our panel does not work very well in that sense. I'm sure bigger companies who have millions for R&D have throttling figured out to an extremely perfect degree, but we're not there yet so the next best thing is how we currently handle abuse.

    I'd like to address all portions of our AUP that you quoted separately.

    High CPU: Customer’s Service (1) cannot burst to 95-100% usage for more than 5 minutes, (2) cannot average higher than 50% usage within a 2 hour period. Gaming plans, services with the high CPU option, and any custom high CPU usage services previously discussed with VirMach may burst to 100% at all times.

    If you want to use all the CPU and bypass fair usage, you can purchase the addon. The choice is yours, and most customers chose the cheaper burstable route, and most customers don't break the AUP. This portion of the AUP is not enforced very strictly, unless you're using a lot of CPU all the time. I'll see if we can change the duration to fall a little more in line with what we actually enforce. I definitely still want to keep the numbers in the AUP more strict than how strictly we actually enforce them, so there's a clear line for customers to attempt to follow with breathing room.

    High Load: Customer’s Service (1) cannot have a 15-minute load average higher than the number of full logical cores assigned, (2) cannot have a 1-day load average higher than 70% of the number of full logical cores assigned.

    This rule of thumb is used for OpenVZ plans and very lenient to begin with and falls in line with us wanting to make sure a server doesn't go above 70% if everyone theoretically wants to use all their allocated resources. In most cases, customers are well over this limit when we take action.

    High Mail Volume: Customer’s Service (1) will have port 25 blocked by default, (2) may send 100 maximum e-mails per hour, (3) must maintain a similar average volume of mail on a week-to-week basis – no bursting. A different arrangement (including some services) may be explicitly discussed with VirMach to override this policy.

    Our lawyer wrote it like this. We don't have port 25 blocked by default, but reserve the right to do so. It also discourages people from using our services for mass mailing and works. I'll see if we can word it better. As for the maximum amount of emails, you can go over that by a bit assuming you're sending personal e-mails. Personal e-mail services such as google have even more strict limits and we'd like to think that if you need a mailserver, you should look elsewhere or at least talk to us first. Bursting e-mails at extreme levels may result in the datacenter's nullroute system kicking in, even if it's a false positive. It also encourages large bulk mailing lists which does result in blacklists if it's spammy.

    High I/O: Customer’s Service (1) cannot have an average IOPS of more than 80 within a 2 hour period, (2) cannot burst to 300MB/s or more disk write average for more than 10 minutes, (3) cannot have more than 300 write operations per second average for more than 1 hour, (4) cannot be above 20% average utilization within a period of 6 hours.

    This is the portion of our AUP that I personally don't really like because it's a little hardr to properly identify and enforce I/O abuse. There's many different factors involved, so we tried to cover all possibly problematic situations as best/simplest we could. I can tell you that our current systems don't really focus on everything written there and don't necessarily enforce all those numbers but it's usually very accurate. The people that end up getting caught by our abuse system are usually already breaking our terms of service by running a program we prohibit (traffic exchange, excessive web macros, etc.)

    (edit) Cloudflare is not letting me submit the portion about network usage for whatever reason but that section of the AUP is pretty self-explanatory.


    Anyway, take everything I said with a grain of salt. Everything that's gone into how we offer our plans and how we have our AUP is based on how I would like to be treated as a customer, how I would like my personal service to perform, and how our team believes it would benefit the majority of customers. People have different preferences, but I wanted to justify/explain some concepts. We're always open to suggestions and improvements.

    I guess my response has turned into what our company specifically does, which is not what I originally intended.

    Thanked by 1Yura
  • raindog308raindog308 Administrator, Veteran

    VirMach said: I wouldn't necessarily like to describe it as "usage is frowned upon."

    I meant it in jest. You were just so specific when others are very vague...it's actually a big positive.

    It's just impossible for the average user to know what their CPU impact would be. Yeah, if you are going to run a bitcoin miner, you should know, but your average bear just doesn't know what is reasonable and how much they're hitting the box.

    Also, if someone sells me a "fair share 2 vcpu" system, why can't I run at 100% all the time? Isn't it the provider's job to nerf me? I can do that on some platforms (Amazon, Azure, DO, etc.) but of course Solus and its kernels don't support this.

    Providers (because of the technology underneath) are good at apportioning disk, network, and RAM, but how much CPU I can use always feels murky to me.

    Maybe I should be a VirMach customer :-)

  • WSSWSS Member

    @raindog308 said:

    VirMach said: I wouldn't necessarily like to describe it as "usage is frowned upon."

    Also, if someone sells me a "fair share 2 vcpu" system, why can't I run at 100% all the time? Isn't it the provider's job to nerf me? I can do that on some platforms (Amazon, Azure, DO, etc.) but of course Solus and its kernels don't support this.

    That's a good way to get told to upgrade or go away. I was given shit before for running apt-get update after being provisioned a slightly older Debian 8. Seriously.. within an hour of it being setup.

    Maybe I should be a VirMach customer :-)

    There are plenty of more expensive hosts out there if you really feel the need to spend more.. or just upgrade the one you want!

  • raindog308raindog308 Administrator, Veteran

    WSS said: That's a good way to get told to upgrade or go away. I was given shit before for running apt-get update after being provisioned a slightly older Debian 8. Seriously.. within an hour of it being setup.

    On...who? I'd mentioned AWS/Azure/DO - was it on one of those!?!? That would really surprise me.

  • WSSWSS Member

    @raindog308 said:

    WSS said: That's a good way to get told to upgrade or go away. I was given shit before for running apt-get update after being provisioned a slightly older Debian 8. Seriously.. within an hour of it being setup.

    On...who? I'd mentioned AWS/Azure/DO - was it on one of those!?!? That would really surprise me.

    Nope. It was a LET host. :)

  • VirMachVirMach Member, Patron Provider
    edited March 2017

    raindog308 said: It's just impossible for the average user to know what their CPU impact would be. Yeah, if you are going to run a bitcoin miner, you should know, but your average bear just doesn't know what is reasonable and how much they're hitting the box.

    That's why we do try to be clear on things that definitely result in abuse (traffic exchange and so on) and we also send warnings explaining the usage before taking action. Only in rare cases so we have to take immediate action if the server is being overloaded.

    In most cases, customers cooperate and we assist them in getting the right plan for what they're attempting to do and/or sometimes help them lower their usage.

    raindog308 said: Also, if someone sells me a "fair share 2 vcpu" system, why can't I run at 100% all the time? Isn't it the provider's job to nerf me? I can do that on some platforms (Amazon, Azure, DO, etc.) but of course Solus and its kernels don't support this.

    As I said we would definitely like to work toward this in the future if/once it's possible. It's obviously a better solution. We're just working with what we have and that might not be as fancy as larger companies.

    raindog308 said: Providers (because of the technology underneath) are good at apportioning disk, network, and RAM, but how much CPU I can use always feels murky to me.

    Maybe I should be a VirMach customer :-)

    I have been in a situation in the past myself (long time ago) where I unknowingly used a lot of processing power that was "fair share" and it resulted in a bad experience for me (and probably the provider), which is why we do have the option to purchase the "high CPU" option that bypasses our fair use policy at a fair price. We do not make any money, in terms of usage, for these "high CPU option" and it's definitely not worthwhile if we're focusing on costs and profits. The option is there so we do not lose customers in instances where they need all the CPU allocated to them, but some people see it as attempting to get more money from them just for the sake of doing it. For example, on our "Enterprise" package we give customers 4 logical cores and price it based on them using an "average" of 2 logical cores, and the pricing comes out to $12. When a customer uses all 4 logical cores all the time, it means that for it to be worthwhile to us, they need to pay us $24 as CPU becomes the bottleneck at that point. However, we only charge $8 for the high CPU option, effectively losing money to keep a customer.

    So I would definitely say being a VirMach customer is actually very nice as you're given both options and can pick the best/cheapest one for your needs. It's better than forcing you to pay for "dedicated" CPU to begin with or having no option to escape the AUP.

    WSS said: I was given shit before for running apt-get update after being provisioned a slightly older Debian 8. Seriously.. within an hour of it being setup.

    If you're referring to our service, we did have a problem with false positives when we were beta testing our script on a small number of servers. I do apologize if you were negatively affected in this isolated incident.

    We do now have a system in place to specifically avoid situations like this triggering our anti-abuse systems.

  • Gamma17Gamma17 Member
    edited March 2017

    VirMach said: The main benefit of virtualization is lost when you want strict lines drawn. By giving you more resources and having a fair use policy, you get more bang for your buck. Providers could give you dedicated resources or limit you specifically to what you pay for, but I don't understand how that benefits you as a consumer.

    Are you really stating that you would rather have only 50% of a core at all times, rather than have the 50% (at the same cost) but also be able to burst to 100% when required?

    Exactly. But I suppose people like to see the higher numbers when they're purchasing the service at a low cost, and then compare it to higher cost services that permit you to use all the resources while criticizing the host for not limiting them. Even if they have the option to purchase more resources and stay within the fair usage policy, they would rather argue. For whatever reason, some people would rather pay $10 for a product with 1x resources than pay $10 for a product with 2x resources that you can use half the time then they like comparing it to the $20 product with 2x resources.

    While burstable resources may sound good, i would almost always prefer strict resource limits, especially for low-end services. While it will make installation process and such somewhat slower, with strict resource limits i can be 100% sure that i will not be suspended at some random moment, and any "burst" in resource usage is, most likely, some issue with my software and is not beneficial to me anyway. And having this limits clearly specified in plan descriptions will be a huge advantage for me.

    In another words yes, if i pay $10 i want to get my 1x resources, not 1x+ 2x burstable, which are not guaranted and can lead to service being suspended.

    Of course i understand that some resources may be hard/impossible to limit with openvz... but still it would be great, at least for KVM.

    And for some reason i suspect that a lot of people feel the same way...

    Thanked by 1deadlyllama
  • WSSWSS Member

    @VirMach said:
    If you're referring to our service, we did have a problem with false positives when we were beta testing our script on a small number of servers. I do apologize if you were negatively affected in this isolated incident.

    No, it wasn't your system.

    I had a technician threaten me in a ticket about "resource abuse" in an unrelated-to-that-vhost ticket. After I politely told them "If updates after provisioning is bad, you're going to have a very unhappy time hosting.", they calmed down.

Sign In or Register to comment.