All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
What do you consider processor abuse, and how do you handle it
This is aimed at the providers, after my own recent experience.
I purchased (year paid) a large LEB, with three CPU's as I have a bit of initial processing required, before things settled down.
I made my script single threaded to lock it to one core, knowing that the SQL backend is likely to tie up the second core. I run the script, things are working well, and my usage is shown as ~2.02.
I figured this would be OK, as it is only using ~66% of the processing power I have purchased.
About 12 hours later, I find my VPS is powered off. Zero warning, nor any sort of notification about the action taken.
Before I talk to my provider about it, I'm curious to see what the general consensus is. Do you look at the individual core usage, or the average of the purchased capacity? Do you warn or notify the customer? If I rerun my script with a nice -n 19, would that be preferred?
Comments
66% of three cores is 100% of two cores. This is usually a lot as CPU us allocated on a fair share basis. Usually this CPU usage (If not affecting others would result in a ticket) if affecting others then a suspension would be the result. Most cases a ticket is sufficient however.
Even if the vps seems to come with 3 cores, I believe they need to be shared with other vps users, not dedicated to you.
Most providers would shut you off, once you start affecting other users. Using 100% of two cores of a 4-core node would not be acceptable by most LEB providers, unless otherwise mentioned or discussed with them.
I do think you should speak with your provider, though.
I understand that I shouldn't use 100% of my allowance, but considering I have paid for 3 cores, what utilisation should I expect to be permitted.
If I had purchased 10 cores, could I then use 2? How is fair share applied in openvz? I really don't mind being given a smaller share of the CPU when it's needed by others, etc
Perhaps another question should be, if I have a small period of processing required, what is the nicest way I can do this? I guess that the nice command means nothing outside of my guest. As timeliness isn't too much of an issue, should I stick a bunch of sleep commands in there to obfuscate my utilisation?
>
The simplest way to put it is: you have paid for access to 3 cores, not to use 3 cores 100% of the time.
I may be naïve here, but are my 2 guest cores directly equivalent to 2 host cores? Surely it is throttled/controlled/oversold/emulated a bit. I had assumed my little slice of CPU core was really only part of an actual core.
Again, I may be completely off here, but i never even considered a 1:1 mapping between guest and host.
Still see my post above asking how I can run this in a friendlier manner
Cheers
SolusVM + OpenVZ doesn't have the ability to do what you've described. It's more or less a "Cloud" feature in terms of resource throttling and scaling.
As defined below. I need to update it to specify concurrently.
CPU Usage:
Subscriber’s programs and services may not use more than 90% of one CPU core per process thread for more than 15 minutes. Subscriber’s programs and services will be arbitrarily “niced” to +19 if found to be using more than 90% of one CPU core per process thread. Subscriber’s programs and services may not use 100% of one CPU core for more than 2 minutes.
I can't think of any LEB provider that sells guaranteed CPU cores.
Damian,
If I nice 19 it myself, is this recognised by the host? Or do you nice 19 my entire VM?
We run a similar policy, except VPSMON manages it for me and just caps CPU.
For OpenVZ, we nice the exact process, and we can see that your processes are nice'd if you choose to do it yourself.
Example:
For KVM, we nice the entire VM, since that's how we see it on our end.
Example:
Either way, we only nice processes that are causing a problem.
It's not that nice to nice the process, but we do something similar too. It helps.
Thanks Damian. I'm in an openvz container, and have already nice 19'ed the script in question.
I hope my provider is as nice (pun fully intended) about it as you are.
For what it's worth, I have made a few changes so that my reported usage is oscillating between 0.90 and 2.00 (out of 3 remember) every minute or so... It would have been much easier if I could limit it to x% and just run the script as was though!
Would providers consider this an adequate response? If not, what else can I do (short of taking my script elsewhere)
@damian so you look at the CPU usage of the process and it it hangs around 100% you nice it or suspend the VM?
If your application supports the correct POSIX signals, https://github.com/opsengine/cpulimit will limit CPU usage as a percentage.
This is one of the reasons I prefer OpenVZ to KVM. You can throttle the amount of allocated CPU by % of available cycles and not just cores like KVM.
People can use 100% of all allocated as far as I'm concerned because it's still only a slice of available and doesn't affect other customers.
Interesting; just checked the source code and found that it uses the SIGSTOP/SIGCONT signals which I myself usually issue by hand when controlling programs in the background. I have two observations:
There's no need for the application to have any special support for POSIX signals; any program (even a simple one-liner main(){while(1){}} loop) is supposed to just work with SIGSTOP and SIGCONT, as the work is done kernel-side, not user-side.
As the program is simply frozen when issued a SIGSTOP and resumed with SIGCONT, I imagine that signaling it to stop whenever it exceeds the CPU usage and then signaling it to continue afterwards could cause some jerky/lagged response, specially in interactive applications; not an issue for batch processes, tho.
I'm interested in how you are using this by hand.
My VPS provider doesn't have the ability to detect the fact that I am running nice 19 and after a script automatically shut down my box a few times, it has finally been suspended.
(Despite all of this, they have been fairly good about it, we have been going backwards and forwards trying to find a way for this to work for the both of us. I am now back up and running with the cpulimit, but not sure how effective it is yet).
However, to me this seems MORE intrusive to the host than running with 'nice 19' as I am still hogging up to x% of my 3 cores, rather than only running on spare cycles.
FWIW: It is a (two actually) PHP5 script(s) querying a mysql database fairly heavily.
Don't use nice. Use ionice. Run your sql backend like the following:
ionice -c 2 -n 7 <your sql backend server>This would slow down your database, causing the php scripts to wait, thus reducing the overall cpu consumption.
Basically I have some long-running processes (daemons) that don't have to be running all the time, but which would be a pain to simply terminate (because their startup would be very slow); so, when I don't need one, I just "kill -STOP" its PID, and when I want it running again, it's "kill -CONT" time.
From the point of your programs, no doubt. From the point of OpenVZ, I'm pretty sure that its task scheduler should do a better job (less intrusive and more efficient) than any userland program doing SIGSTOP/SIGCONTs... so +1 for ' nice 19' instead of 'cpulimit'. The only advantage of the latter would be to be work with percentages, which "nice" doesn't.
There is also the problem of io wait. That is using cpu cycles too.
Also, on a massively threaded environment such as OVZ, context switching is done very aggressively. We have on the larger servers thousands of threads running on 24 cores and sharing the CPU cores is a problem in itself, adding extra checks with priorities and counting shares will not help the issue. It does work, but any more work on the cpu in order to share more evenly due to abuse will only increase the total load in the end.
The way load goes from an app, single-threaded or not, depends a lot on many factors, due to this, some are munching a lot more CPU than others showing similar loads. OVZ is not well isolated and the massive threading does not help things.
Our policy is to allow a maximum of load equal to the number of cores, but this means we cant put many small VPSes on a big server, small VPSes with once vCPU on E3, big ones with multiple vCPUs on Dual E5 so the number of threads is limited per core in order to avoid the crazy context switching going even crazier, to the point the CPU does almost exclusively that. Also, the maximum load does not mean can be the average load too, locking cores/threads is also not allowed, even if the plan has, say, 4 cores and one is locked by a single-threaded app, we cannot allow that.
OpenVZ is such a massive multi-threaded mess. Honestly, KVM is a much better platform to run CPU Intensive apps/scripts on. The effect on other containers is SIGNIFICANTLY reduced.
Better for the the customer to hog CPU...at the expense of other customers. Not sure what you mean by "multi-threaded mess". The only bottleneck I know about is file system journal and you can use ploop to get around that now. All depends on what you are doing. I never have more than a few dozen people on a box and I don't have any problems with it.
I also use KVM so I don't have an bias btw. I find KVM is more resource intensive all around so I don't like to use it for any of my own stuff. I also find it is more spikey on the load of the server. Not sure if there is a technical term for that. It just doesn't share resources as gracefully. I just offer it for anyone who wants to use it instead.