Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Advertise on LowEndTalk.com
Unable to find cause of high disk I/O
New on LowEndTalk? Please read our 'Community Rules' by clicking on it in the right menu!

Unable to find cause of high disk I/O

emmd19emmd19 Member
edited February 2017 in Help

I've been experiencing issues with high disk I/O on my KVM VPS recently, which has caused several excessive I/O alerts from my host (a well-known provider here on LEB/LET). In particular, there is always a constant disk read of about 7MB/s. The VPS is running Ubuntu 16.04 64-bit and has 1.5GB of RAM, of which about 2/3 is in use at any given time. Swap usage hovers between 25-100MB out of 1.2GB.

Here's the output from a single invocation of iostat, which, according to my understanding, represent running averages since boot:
Here's the I/O graph over the last 24 hours from my host's control panel:

The VPS is used for web/HTTP and PostgreSQL database. I initially suspected the Postgres database was causing issues, but I can't seem to find the source of this phantom read I/O. iotop does not provide any clues, and running iostat at 1s intervals show that disk I/O is minimal. Furthermore, vmstat shows that swap activity is minimal as well. I even considered the possibility that my host's I/O metering was buggy, but they replied that since I/O usage is read directly from the hypervisor it is impossible for their readings to be wrong.

I'm at my wit's end trying to figure this out. Does anyone have any ideas?

«1

Comments

  • I'd run lsof and start checking processes. There's far too much information which you just haven't given us. What sort of host is this running on? What version KVM? Etc, etc..

    There may very well be a nasty way VirtIO is being handled with your Postgres DBs. Check your systat and everything else.

  • emmd19emmd19 Member
    edited February 2017

    How would lsof help? Since I'm just a client I have no idea about the particulars about my hosts underlying KVM implementation, sorry :(

  • Crawl through your dmesg, look for "virtio" information; what's your ethernet device show as, etc..

    What's your build? What're you running for these services- are they stock distrib, custom, etc..

    To check and change things without taking down services, you can play around with ionice.

  • emmd19emmd19 Member
    edited February 2017

    lspci:
    00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01) 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 00:03.0 Ethernet controller: Red Hat, Inc Virtio network device 00:04.0 SCSI storage controller: Red Hat, Inc Virtio block device 00:05.0 SCSI storage controller: Red Hat, Inc Virtio block device 00:06.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon

    All packages are stock - nginx, postgres from Ubuntu repos, and some custom Python/Django projects running behind gunicorn. These are not resource-heavy at all, with the possible exception of frequent database accesses (although certainly not to the point of several MB/s sustained). I've just remounted / with noatime - will let you know how that goes...

  • Your iostat screenshot above shows I/O that seems to be consistent with the graph. When you say minimal I/O with iostat at 1s intervals, does it look like the screenshot above?

    Have you tried stopping Postgresql and measuring the I/O?

  • IshaqIshaq Member, Provider

    What does iotop show?

    [BudgetNode] DDoS Protected. 7 Locations (US/EU). Check out our latest offer!
  • @xyz Nope, when running iostat at 1s intervals most of them look like this:

  • IshaqIshaq Member, Provider

    Your TPS value is high especially if the provider uses HDDs.

    [BudgetNode] DDoS Protected. 7 Locations (US/EU). Check out our latest offer!
  • emmd19emmd19 Member
    edited February 2017

    @Ishaq When running iotop, the disk read is usually anywhere from 0 to a few hundred KB/s. There are rare moments when it spikes/bursts due to a Postgres SELECT running on a large table, however, these never last more than a second or two. Basically there's nothing I can see that would explain sustained disk I/O.

  • @Ishaq said:
    Your TPS value is high especially if the provider uses HDDs.

    I guess there's not much point in hiding my provider lol. It's LunaNode, and IIRC this is one of their SSD-cached plans out of OVH-BHS.

  • IshaqIshaq Member, Provider

    Try installing and running atop.

    [BudgetNode] DDoS Protected. 7 Locations (US/EU). Check out our latest offer!
  • Will try that now. In the meantime, here's about 30 seconds worth of vmstat 1 in case that helps:

  • emmd19 said: @xyz Nope, when running iostat at 1s intervals most of them look like this:

    Could you run iostat 10, leave it for a minute, and screenshot all of the output?

  • atop -d -A specifically. I'm so used to systat that I had to look that up. :D

  • @Ishaq @WSS atop -d -A:

  • IshaqIshaq Member, Provider

    How strange.

    [BudgetNode] DDoS Protected. 7 Locations (US/EU). Check out our latest offer!
  • Huh. What's throwing me for a loop is that it's showing 20% USER above, but then- nada, so we should be seeing something here.

    I'm just going to assume it's an ancient KVM on CentOS 6.

    I'm assuming you've tried changing priorities and nonsuch.

  • emmd19emmd19 Member
    edited February 2017

    @xyz iostat 10 for 1 minute (1st entry is the average since boot):

  • @WSS My CPU load average is around 20-30% - is that what USER% means?

  • xyzxyz Member
    edited February 2017

    CPU usage is user+system (+nice if you have any nice'd processes).

    From that iostat, it looks like to me that your I/O is usually low, but you get spikes (like that 2047 tps reading), which causes the average to be what it is. The graph is likely averaging over a long period of time, and the first iostat reading shows an average which seems to be in-line if you average all your other iostat readings.

    Thanked by 1WSS
  • @xyz said:
    CPU usage is user+system (+nice if you have any nice'd processes).

    From that iostat, it looks like to me that your I/O is usually low, but you get spikes (like that 2047 tps reading), which causes the average to be what it is. The graph is likely averaging over a long period of time, and the first iostat reading shows an average which seems to be in-line if you average all your other iostat readings.

    The problem with this is the generated graph, because it looks pretty consistent, I guess we'd need to have better sample data from what the host is running- and again- would be a lot more useful from the host perspective than the QEMU. The end result is that we're all left wondering. :)

  • @xyz said:
    CPU usage is user+system.

    From that iostat, it looks like to me that your I/O is usually low, but you get spikes (like that 2047 tps reading), which causes the average to be what it is. The graph is likely averaging over a long period of time, and the first iostat reading shows an average which seems to be in-line if you average all your other iostat readings.

    Hmm...that makes sense I suppose. I guess the short-term is to upgrade to beefier hardware...

  • @emmd19 said:
    @WSS My CPU load average is around 20-30% - is that what USER% means?

    This little article should help you understand what the different numbers actually mean: http://blog.scoutapp.com/articles/2009/07/31/understanding-load-averages

  • So going off the conclusion that I need to upgrade my infrastructure - what do you guys recommend? Is this level of resource/disk utilization still within the realm of VPS, or is dedicated the way to go?

  • I'd recommend you start benchmarking/setting up accounting to see if you can actually find what's going on, first. If you're having quick little stabs that even out to 7MB/s, you might just have to work on your queries and/or change the design. We're all flying blind here- you might try asking your host to show you your allocated system use as well as running accounting under your processes.

  • Your iostat shows up to 25 MB/s read. At the same time it shows that the amount of data read per minute is about the amount one would expect per 10s. Ergo you have something that reads a lot in spikes.

    You will need to watch with finer granularity and find out who is reading cyclically from vda.

    Also show your mounts and tell us about your swap.

    My favourite prime number is 42. - \forall cpu in {intel, amd, arm}: cpu->speed -= cpu->speed/100 x irandom(15, 30) | state := hacked

  • @bsdguy said:
    Also show your mounts and tell us about your swap.

    Filesystem wouldn't hurt, either. You brazen hussy.

  • Can't you ever think about anything else, you slut? How perfect!

    File system? Don't care yet. 25 MB/s smells strongly like cache. A propos smelling: That whole thing smells.

    My favourite prime number is 42. - \forall cpu in {intel, amd, arm}: cpu->speed -= cpu->speed/100 x irandom(15, 30) | state := hacked

  • Being that we don't have any host specs, I still wonder if we're getting a combination of random select hits and just an overall shitty driver base since it's all virtio.

    I know you'll do such crazy things for 25MB/s.. even if your sisters' eyebrow entrances me so.

  • emmd19emmd19 Member
    edited February 2017

    All right guys, get your mind out of the gutter :P Filesystems are nothing special, just a single 15GB / formatted as ext4 with plenty of free space:
    Filesystem Size Used Avail Use% Mounted on udev 744M 12K 744M 1% /dev tmpfs 150M 1.3M 149M 1% /run /dev/vda1 16G 9.1G 6.1G 60% / none 4.0K 0 4.0K 0% /sys/fs/cgroup none 5.0M 0 5.0M 0% /run/lock none 749M 0 749M 0% /run/shm none 100M 0 100M 0% /run/user

    Swap consists of a ~256MB swap on /dev/vdb1 and an additional 1GB swapfile mounted on /dev/vda1.

    Swap utilization is modest and currently at 218/1061MB with minimal swap activity.

  • WSS said: The problem with this is the generated graph, because it looks pretty consistent

    The graph looks like it takes 5-6 samples per hour, so it's likely averaging every ~10 minutes. If the load is consistent for those periods of times, you're not going to see spikes that you see with 10s granularity.

    emmd19 said: I guess the short-term is to upgrade to beefier hardware...

    As others mentioned, see if you can identify what's causing the spikes. Are these perhaps the large SELECTs you're referring to?

    You can also just stop PostgreSQL temporarily and do a similar iostat test to see if the read spikes disappear.

    Disk is usually the bane of most server workloads (unless you're on SSDs) - if you do look at upgrading, perhaps consider SSD storage or more RAM (to cache reads more, assuming that they're cacheable).

  • Local-file-as-caches. That'll do. Throw more RAM at it.

  • emmd19emmd19 Member
    edited February 2017

    I've done a brief review of the application code that's running (it powers a website that gets about 200K-400K hits a month). Unfortunately there's not much optimization that can be done in terms of DB calls; the app makes frequent calls to an external API for each incoming HTTP request from a user, and the response of each call results in accesses to many individual rows. Hence it's an inherently read-heavy (and to a lesser extent write-heavy) app.

  • Throw RAM at it and let it become someone else's eventual problem. :D

    Thanked by 1netomx
  • @xyz said:

    WSS said: The problem with this is the generated graph, because it looks pretty consistent

    The graph looks like it takes 5-6 samples per hour, so it's likely averaging every ~10 minutes. If the load is consistent for those periods of times, you're not going to see spikes that you see with 10s granularity.

    emmd19 said: I guess the short-term is to upgrade to beefier hardware...

    As others mentioned, see if you can identify what's causing the spikes. Are these perhaps the large SELECTs you're referring to?

    You can also just stop PostgreSQL temporarily and do a similar iostat test to see if the read spikes disappear.

    Disk is usually the bane of most server workloads (unless you're on SSDs) - if you do look at upgrading, perhaps consider SSD storage or more RAM (to cache reads more, assuming that they're cacheable).

    I can't stop PostgreSQL since it is somewhat mission-critical right now, but I can clone everything to a new VM and let it sit for a while (where it won't be dealing with any workload). I'll let you know how that goes.

  • bsdguybsdguy Member
    edited February 2017

    Your web server (and hence db) isn't guilty I think. The hits don't explain that.
    Before we go on, turn off your swap at least on vda. Having a cache of about 1/3rd of total mem and having a swap is not needed anyway.

    And show us an lsof output cleaned of standard shit.

    Thanked by 1WSS

    My favourite prime number is 42. - \forall cpu in {intel, amd, arm}: cpu->speed -= cpu->speed/100 x irandom(15, 30) | state := hacked

  • emmd19 said: I've done a brief review of the application code that's running (it powers a website that gets about 200K-400K hits a month). Unfortunately there's not much optimization that can be done in terms of DB calls; the app makes frequent calls to an external API for each incoming HTTP request from a user, and the response of each call results in accesses to many individual rows. Hence it's an inherently read-heavy (and to a lesser extent write-heavy) app.

    10k reqs/day is around one every ~9 seconds. Do you get clumps of requests together often? The graphs show fairly steady load throughout the day (with a few peaks here and there), but you're seeing very spiky load during smaller windows. It's hard to guess, but if you have consistent load, you shouldn't be getting those big read spikes - you'd expect it to be spread out more.

    In other words, your load may be coming from another source.
    Then again, I don't know your application...

  • A very simple means to track this would be to do a ps/systat(atop)/etc from a script to read through and see just what's going on. At least with a basic ext4 system it isn't going to be a silly journal issue or anything like that. Basically, it just needs to be babysat and watched.

  • A very simple means to track this would be to do a ps/systat(atop)/etc from a script to read through and see just what's going on. At least with a basic ext4 system it isn't going to be a silly journal issue or anything like that. Basically, it just needs to be babysat and watched.

  • eva2000eva2000 Member
    edited February 2017

    emmd19 said: I've been experiencing issues with high disk I/O on my KVM VPS recently, which has caused several excessive I/O alerts from my host (a well-known provider here on LEB/LET). In particular, there is always a constant disk read of about 7MB/s. The VPS is running Ubuntu 16.04 64-bit and has 1.5GB of RAM, of which about 2/3 is in use at any given time. Swap usage hovers between 25-100MB out of 1.2GB.

    if you have sysstat and sar installed and configured, you'd have historic records for close to exact date and time (10 min interval based entries by default) so you can go back to that exact date/time to review sar -d output to at least see which block device it's coming from with help of lsblk output matching

    last 24hrs for centos not sure for Ubuntu where you sar logs are

    sar -d -f /var/log/sa/sa$(date +%d -d yesterday)

    * Centmin Mod Project (HTTP/2 support + ngx_pagespeed + Nginx Lua + Vhost Stats)
    * Centmin Mod LEMP Stack Quick Install Guide
  • My guess (entirely based on the limited context and totally based on previous hunches) is that it's the DB. There are probably tables that could do with an index update and/or an overall statistics update (and related general DB maintenance). When was the last time you explicitly ran some PG maintenace scripts?

    Easy way to test (if possible) - shutdown the DB for a bit and see if the IO graph drops (consistently). Of course it is a HeisenTest(tm) but it's probably worth it. Needless to say I assume you're not going to have some logging daemon spew lots of messages, promptly increasing the IO back to where it was.

    Thanked by 1Falzo
  • rincewindrincewind Member
    edited February 2017

    Install the blktrace package, and run

    btrace /dev/vda1
    

    for an online trace.

    You can also use blktrace to record logs and parse them later with blkparse.

  • emmd19emmd19 Member
    edited February 2017

    @nullnothere Yep, shutting off Postgres makes disk I/O drop like a rock. What sort of maintenance do you recommend? I just finished reindexing the entire database, and so far I've just been letting autovacuum do its thing.

  • @emmd19 said: Yep, shutting off Postgres makes disk I/O drop like a rock. What sort of maintenance do you recommend?

    I'm no PG expert (hopefully some other PG Guru's here can help) but if you're doing all the usual good things (vacuum, analyze, reindex) then I think you're OK. Check that they are happening frequently enough (or that they're happening correctly). You may need to increase autovacuum frequency as well.

    If despite these maintenance jobs (etc.), you're having high IO then you'll have to start looking at the queries (figure out the slow ones etc.). Perhaps somewhere you're doing a table scan (missing an index). The nice thing is that it doesn't appear to be a CPU hog (so it's not some sort of a stupid loop join etc. that the query planner is picking for whatever reason).

    You may have "grown" (DB wise) to a point where now you need more memory for optimum performance etc.

    Hope this gets you going (you're now entering DBA land...)

  • Turns out that I had some relatively DB-intensive cron jobs that were originally configured to run every minute. This was fine when the DB was small but not so much now that the DB is much larger (~2GB). I've set them to run less frequently and that seems to have done the trick for now!

    Thanked by 1Falzo
  • @nullnothere I did some more digging and found a query that is essentially a table scan (filtering on an unindexed boolean). Added a partial index on that field and so far so good.

    Thanked by 1nullnothere
  • As you are at it you might want to considerably increase index memory for PG rather than wasting it on disk caching.

    My favourite prime number is 42. - \forall cpu in {intel, amd, arm}: cpu->speed -= cpu->speed/100 x irandom(15, 30) | state := hacked

  • @bsdguy said:
    As you are at it you might want to considerably increase index memory for PG rather than wasting it on disk caching.

    WSS said: ... Throw more RAM at it.

    WSS said: Throw RAM at it ...

    ;-) ;-) ;-)

    Thanked by 1nullnothere

    UltraVPS.eu KVM in US/UK/NL/DE: 15% off first 6 month | Netcup VPS/rootDS - 5€ off: 36nc15279180197 (ref)

  • @Ishaq said:
    What does iotop show?

    Thanks for the advice.

  • I used to have a script somewhere for measuring the actual IO Disk R+W requests p/second(s) on a per process basis, this thread reminds me I should probably share that.

    Thanked by 1aboanas93

    I am no longer active here, find me at https://talk.lowendspirit.com (Just like LET without the scams)

Sign In or Register to comment.