Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Where is the bottleneck ?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Where is the bottleneck ?

My server is running slow, yet the cpu load shows nothing big.
Can someone explain to me what's causing my website slow, where is the bottleneck ? and how to check it

«1

Comments

  • NeoonNeoon Community Contributor, Veteran

    run top and post that

  • tetechtetech Member

    @yokowasis said:

    My server is running slow, yet the cpu load shows nothing big.
    Can someone explain to me what's causing my website slow, where is the bottleneck ? and how to check it

    Probably I/O, vmstat or iotop.

  • lanefulanefu Member

    Yeah probably an io component to start.

    Other possibilites

    • mysql isn't tuned for your resources. Its memory footprint seems small
    • bad query design missing indexes
    • php hasn't been tuned for your resources
    • missing noatime and nodiratime mount options for your DB volume
    • php add need some sort of cache like redis or memcache
    Thanked by 2o_be_one tiendungdev
  • enable slowlog on php-fpm config

  • jsgjsg Member, Resident Benchmarker

    @yokowasis

    Well for a start a load avg. of 18 on a 12 cores system isn't exactly low.

    Plus have a look at the IO too, both disk and net.

  • @jsg said:
    @yokowasis

    Well for a start a load avg. of 18 on a 12 cores system isn't exactly low.

    Plus have a look at the IO too, both disk and net.

    I am referring to the processor graphs. No core seems utilized at 100% yet the load avg is very high.

  • tetechtetech Member

    @yokowasis said:

    @jsg said:
    @yokowasis

    Well for a start a load avg. of 18 on a 12 cores system isn't exactly low.

    Plus have a look at the IO too, both disk and net.

    I am referring to the processor graphs. No core seems utilized at 100% yet the load avg is very high.

    It takes into account I/O wait.

  • jsgjsg Member, Resident Benchmarker

    @yokowasis said:

    @jsg said:
    @yokowasis

    Well for a start a load avg. of 18 on a 12 cores system isn't exactly low.

    Plus have a look at the IO too, both disk and net.

    I am referring to the processor graphs. No core seems utilized at 100% yet the load avg is very high.

    Don't put too much significance into those graphs. Look at the numbers, not at the eye candy.

    Two quick side notes:
    (a) are you sure you want swap active?
    (b) your setup uses only a very small part of the plenty memory you have. You might want to think about trading processor load for memory. I suggest to considerably extend your caching (on all levels, from mysql up to web pages).

    Thanked by 2bulbasaur kkrajk
  • This is definitely an I/O issue and tetech rightly pointed out that load average includes I/O. It's a measure of how long processes are waiting for a chance to run.

    Post an iotop, really difficult to help you otherwise.

  • varwwwvarwww Member
    edited June 2021

    Increase innodb_buffer_pool_size to about 15G and restart mysql server

    Thanked by 1lanefu
  • PUSHR_VictorPUSHR_Victor Member, Host Rep

    Your processes are going to D state, which usually means they are fighting for IO. In other words, your disks are overloaded.

    Thanked by 2o_be_one TimboJones
  • You guys were right, it was an io issue. I added 2 more nvme with raid-0. The problem is solved.

    @jsg said:
    Two quick side notes:
    (a) are you sure you want swap active?
    (b) your setup uses only a very small part of the plenty memory you have. You might want to think about trading processor load for memory. I suggest to considerably extend your caching (on all levels, from mysql up to web pages).

    Thanks Noted.

  • raindog308raindog308 Administrator, Veteran

    @yokowasis said: 2 more nvme with raid-0

    fast-forward a month

    "Guys, how can I restore my system from backup?"

  • LeviLevi Member

    Mysql_tunner.pl that sql server. Apply recommendations. Second, try to adjust php-fpm settings. If you have nvme, use more disk cache.

  • ericlsericls Member, Patron Provider

    That load is very high for that CPU usage... looks like a lot of time is waiting on IO

  • jsgjsg Member, Resident Benchmarker

    Hmmm, if a (reasonable size, reasonable quality) NVMe isn't good enough than one has an app with rqs in the tens of thousands - and/or one has an application design and a caching problem and that should be addressed. "moar nvmes!" usually doesn't solve the problem but rather covers the symptoms and buys some time. Just saying ...

    Thanked by 1chihcherng
  • yokowasisyokowasis Member
    edited June 2021

    @raindog308 said:

    @yokowasis said: 2 more nvme with raid-0

    fast-forward a month

    "Guys, how can I restore my system from backup?"

    YOLO.

    So long the nvme doesn't break, it should be fine, right ? I have faith in hetzner nvme. Most of the data isn't that important anyway. They all got deleted once a month.

    @jsg said:
    Hmmm, if a (reasonable size, reasonable quality) NVMe isn't good enough than one has an app with rqs in the tens of thousands - and/or one has an application design and a caching problem and that should be addressed. "moar nvmes!" usually doesn't solve the problem but rather covers the symptoms and buys some time. Just saying ...

    It's AX41. The default setup is 2 x 2TB HDD with RAID1. Since it's just mainly wordpress sites with a total under 1K page views / day, and most of the time, the server is just idling. I though it's fine.

  • jsgjsg Member, Resident Benchmarker

    @yokowasis said:
    It's AX41. The default setup is 2 x 2TB HDD with RAID1. Since it's just mainly wordpress sites with a total under 1K page views / day, and most of the time, the server is just idling. I though it's fine.

    Sorry, no, it's absolutely not fine when such a beast dedi (6 core Ryzen 3000, 64 GB memory) with a paltry less than 1k page views per day gets down to snail speed and/or up to high CPU load. There's either something wrong with your box or config or you have had hackers visiting your system/site.

    And btw. before shelling out more money for NVMes you really should think about my advice and use what you already have (plenty memory) and what is way faster than NVMes too.

    Frankly, I think you should also inspect your system and do your best in terms of looking for rootkits, malware, etc.

  • isunbejoisunbejo Member
    edited June 2021

    nginx php-fpm, dynamic content data on ramdisk 32GB (tmpfs)
    1200 - 1500 req/s

  • webdevwebdev Member

    1K page views / day, $15 / y VPS is enough for that...

    Thanked by 1o_be_one
  • Your WP page full of bloatware..

  • @webdev said:
    1K page views / day, $15 / y VPS is enough for that...

    The page views is 1k at the same time as in 1k visit in 5 a seconds, not through the day. And you know the human. If a page doesn't load within 5 sec, they will reload the page. Which makes it even worse.

    Tried with a normal vps and shared hosting. It doesn't work.

    @jsg said:

    @yokowasis said:
    It's AX41. The default setup is 2 x 2TB HDD with RAID1. Since it's just mainly wordpress sites with a total under 1K page views / day, and most of the time, the server is just idling. I though it's fine.

    Sorry, no, it's absolutely not fine when such a beast dedi (6 core Ryzen 3000, 64 GB memory) with a paltry less than 1k page views per day gets down to snail speed and/or up to high CPU load. There's either something wrong with your box or config or you have had hackers visiting your system/site.

    And btw. before shelling out more money for NVMes you really should think about my advice and use what you already have (plenty memory) and what is way faster than NVMes too.

    Frankly, I think you should also inspect your system and do your best in terms of looking for rootkits, malware, etc.

    Are you suggesting I am moving the database to ram? The WordPress has very specific use case. 1k visitor at the same time, and all of them write to database at the same time. An image uploader website if you could say.

    What's the normal writing speed of RAID - 1 spinning HDD? Is RAID - 1 could make the performance worse for write intensive website? I mean the file get written into 2 place at the same time.

  • @chocolateshirt said:
    Your WP page full of bloatware..

    It's a custom made theme. The function mainly for uploading article and a lot of images. The article will then be sent to Google firestore, the image will be sent to another cdn.

    It doesn't serve the visitor directly. So nothing fancy or plugin going on.

    When I said 1k page views, it's a 1k of websites Admin doing stuff. Not normal visitor.

  • TimboJonesTimboJones Member
    edited June 2021

    @yokowasis said:

    @chocolateshirt said:
    Your WP page full of bloatware..

    It's a custom made theme. The function mainly for uploading article and a lot of images. The article will then be sent to Google firestore, the image will be sent to another cdn.

    Are you actually writing images to a database or just writing a reference to an image on the filesystem?

    It doesn't serve the visitor directly. So nothing fancy or plugin going on.

    When I said 1k page views, it's a 1k of websites Admin doing stuff. Not normal visitor.

    Well, that's not helpful since that's not what page views means.

    Basically, people can probably improve the efficiency of your setup at every step if you provide enough details.

    Doing something repetitive in a 5 second window type activity is begging for RAMDISK. And you got tons.

    You also should have SSD, not HDD'S. How much space do you actually need? You'd probably be better off using a single NVMe and daily backup to local HDD.

  • EvoxtEvoxt Member
    edited June 2021

    Has to be IO. Do a IOPS check. Also do some caching, that will help a lot

  • @TimboJones said:

    @yokowasis said:

    @chocolateshirt said:
    Your WP page full of bloatware..

    It's a custom made theme. The function mainly for uploading article and a lot of images. The article will then be sent to Google firestore, the image will be sent to another cdn.

    Are you actually writing images to a database or just writing a reference to an image on the filesystem?

    Just a reference. But admin can upload 100 images at one batch, or no images at all.

    It doesn't serve the visitor directly. So nothing fancy or plugin going on.

    When I said 1k page views, it's a 1k of websites Admin doing stuff. Not normal visitor.

    Well, that's not helpful since that's not what page views means.

    Basically, people can probably improve the efficiency of your setup at every step if you provide enough details.

    Doing something repetitive in a 5 second window type activity is begging for RAMDISK. And you got tons.

    You also should have SSD, not HDD'S. How much space do you actually need? You'd probably be better off using a single NVMe and daily backup to local HDD.

    Yeah that's where is was wrong. I though HDD is enough. Because the website is mostly used by admin, not user. It works fine since January.

  • pbxpbx Member

    Throwing more hardware is never the right long term solution.

    Optimal use of your RAM, using several cache layers and tuning your code and config files is better in the long run. If you are not using your server to its full potential, why bother upgrading?

  • @pbx said:
    Throwing more hardware is never the right long term solution.

    Sometimes it's THE only solution. it's called scaling. No matter how much you optimize there is so much you can do. In my case is a spinning HDD.

    No matter how much I optimize the RAM for cache, there is not much you can do if the purposes of apps is to to upload a bunch of files.

  • jsgjsg Member, Resident Benchmarker
    edited June 2021

    @yokowasis said:
    Are you suggesting I am moving the database to ram? The WordPress has very specific use case. 1k visitor at the same time, and all of them write to database at the same time. An image uploader website if you could say.

    Nope, I wasn't yet suggesting anything concrete, because I know very little about your use case and setup/config. But one thing is clear: if your AX41 is very slow plus it seems to be IO related plus you have tons of unused RAM you're doing something wrong.

    What's the normal writing speed of RAID - 1 spinning HDD? Is RAID - 1 could make the performance worse for write intensive website? I mean the file get written into 2 place at the same time.

    No, Raid 1 (even software Raid 1) has no noticeable performance impact.

    Some remarks based on info presented since my last post:

    • @TimboJones is right, you should think and think hard whether you really want to store the (presumable not tiny) images in the DB itself. Usually it's better to just put a reference (e.g. file name) into the DB.
    • 1k writes/sec shouldn't be a problem at all for any DB, even sqlite, unless of course your structure (e.g. MB size images in some table) is, uhm, strange.
    • Can you use spindles and what's their speed? Depends on usage; writing tons of chunks all over the place is quite different from writing a few large chunks serially. As a general rule of thumb you can assume that an HDD can write about 80 MB/s. But careful, that's just a crude ballpark number and it depends on quite a few factors like OS, frequency of writes, chunk size, total size, etc.Also keep in mind that the numbers given for SSDs and even NVMEs (like "1.5 GB/s!") are almost always based on some assumptions, basically the ideal case - which can be quite far off a particular use case. For less optimal situations the actual write performance of an SSD can be more HDD like.

    We can't really help you/provide good concrete advice with the very limited info piecemeal you provide. So, if you want to get good and useful advice for your situation you must tell us much more.

    Thanked by 2pbx chihcherng
Sign In or Register to comment.