New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Generic way not to hammer disk and io
Suppose I wish to run a cpu or io intensive job on my vps. Eg zip 1 million files, or it could be anything. But I do not care if it takes much longer to finish (E.g. 10 or 20 times slower than usual) just as long as I don't hammer resources. I am willing to wait even if it takes days. Is there a generic way to do that?
Comments
Nice?
https://en.wikipedia.org/wiki/Nice_(Unix)
Edit - I think CPULimit is more of what you're looking for;
https://www.cyberciti.biz/faq/cpu-usage-limiter-for-linux/
How about limiting disk?
You have ionice - but like nice, it's more about setting priority.
maybe sleep 1 between the files that you're zipping?
I am looking for generic. I am running many resource intensive tasks
But for zip, how do you put sleep between files?
Say I am doing "zip -r 1.zip test" where test is a directory that contains 1 million files?
Nope, the sleep method would only work if you would create a separate zip file per source file.
We used cgroups with great success for similar tasks (both cpu and io can be limited)
https://www.kernel.org/doc/Documentation/cgroup-v1/blkio-controller.txt
Nice is processing priority if I have not mistaken
Ionice for io priority.. i.e reads and writes
if there's something else even better, I would love to know!
You could use for example a simple python script to loop over all the files and add them individually while sleeping between each file.
https://unix.stackexchange.com/questions/81433/high-cpu-utilization-with-zip-command
This cpulimit application could help if it works as described.
Nice creative alternative but there is a possibility it might still hammer the disk during the file operation being done... but definitely something to look into for its simplicity
What you want to do in summary is to not overlap I/O intensive tasks so that disks don't do more work than they'd normally do. The only easy way to do that is to serialize them. There are plenty of ways that are a bit more work, but they depend on your machine and the kind of tasks you're running.
For instance, if you have plenty of memory (much more than the resulting size of your zipped files), you could make the destination of your zips a ramdisk so you're only doing reads during the zips, then you could copy the zips at the end so you're only doing writes.
If you're doing lots, you could just do zip to ramdisk and copy zip to disk in a loop.
Another thing you could do is run a script while zipping to pause the process when you get more than a certain number of I/O operations per second. This is pretty easy if your OS has an iostat command.
I will try nice and ionice, thank you for advices.
zip is just a sample, it is not the only thing that I need to run. I have many programs that I did not write and can not rewrite.
If your provider is monitoring your CPU/IO usage, and you don't have contention on the node, then nice/ionice won't prevent you from getting suspended by the provider, because your process will still be hammering the node. Nice only sets relative priority; if there's nothing else competing for the CPU, you'll still get 100% of the CPU.
cpulimit might be a better option; essentially, it monitors top/iotop and suspends (a la ctrl-Z) the process until the load comes down.
cgroups should be an even more appropriate solution; I've never tried that.
If I am running in KVM, then it might have no effect right? Since I have no other programs running other than my batch program
Yes, if there's only one process active, then nice/ionice won't do much. I'm not sure what would happen under OVZ, but probably the same.
what about running docker https://docs.docker.com/engine/admin/resource_constraints/
docker gives a lot of control for resource constraint. docker does have good control of cpu usage managing https://stackoverflow.com/questions/33423164/docker-container-cpu-allocation
I haven't used Docker yet (on my to-do list!), but again this would all be within the VM (nested containerization). @jcaleb only needs to run one job on his VM, so I'm not sure multiple containers would help.
The host can certainly limit the VM's usage, e.g. via Proxmox config.
I am just trying to avoid being suspended or something. But can't afford a dedicated for now.
I think cpulimit should suit your use case. It may be worth asking your provider for input.