Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Advertise on LowEndTalk.com
Best Practice for Serving a High Loaded (700-800 Concurrent Requests) Website
New on LowEndTalk? Please Register and read our Community Rules.

Best Practice for Serving a High Loaded (700-800 Concurrent Requests) Website

Mahfuz_SS_EHLMahfuz_SS_EHL Member, Provider

Hello,

A website built on WordPress, concurrent hits/requests about 700-800 (shows in LiteSpeed WebAdmin), what is the best practice to serve it ?? I assume the following:

Core: 8-10 Core Processor (Single Thread Performance should be good)
Ram: Not less than 32 GB
WebServer: LiteSpeed
Disk: NVMe SSD
MySQL Max Connection: 500-1000
With increasing php_max_connection

I want to know what you would do in this situation ? What kind of server, what kind of setup etc.

Thanks.

AlphaSSL Revocation Issue is being investigated.

«1

Comments

  • You already gave the server specs, so what are you looking for?

    Thanked by 1Mahfuz_SS_EHL
  • Mahfuz_SS_EHLMahfuz_SS_EHL Member, Provider

    @tetech said:
    You already gave the server specs, so what are you looking for?

    Yes, I think I'm pretty much sure of a server specs but when I tried, I saw I get MySQL (using MariaDB) going down repeatedly or performing slowly.

    Thanked by 1postcd

    AlphaSSL Revocation Issue is being investigated.

  • If this is standalone setup (without any panels etc..) I could help.. I am serving more on similar hardware under LXC.. It's not Wordpress tho.. it's Laravel but Wordpress is having tons of "caching plugins" so should not be issue

    Thanked by 1Mahfuz_SS_EHL

    I can do: Gentoo & Debian with security and performance in mind

  • Mahfuz_SS_EHLMahfuz_SS_EHL Member, Provider

    @amarc said:
    If this is standalone setup (without any panels etc..) I could help.. I am serving more on similar hardware under LXC.. It's not Wordpress tho.. it's Laravel but Wordpress is having tons of "caching plugins" so should not be issue

    I could do without any panel but the client needs something with UI so that they can manage easily, that's the issue. Though I'm not sure if they have plugins enabled or not.

    AlphaSSL Revocation Issue is being investigated.

  • Budget and current setup? Also, I believe LiteSpeed has LSCache or something similar for WordPress, perhaps you should try that.

    Thanked by 1Mahfuz_SS_EHL
  • whats the bottleneck from observation?
    Do you already put cache somewhere?

    that server can handle the traffic, but without more information about your setup I only can assume that

    Thanked by 1Mahfuz_SS_EHL

    We only support unsupported OS!

  • Calling in the expert @eva2000

  • Varnish and memached. Should suffice but will add serious challenge to administration.

    hostwp.net -- Wordpress Hosting for Developers.

  • Don't hit PHP when possible, serve static content. LSCache or W3 Total Cache will work.

    I used to run Varnish in front of the web server. A 1 core with 512MB was enough to handle more than 2500 concurrent visitors (according to Google Analytics, not requests). Varnish was almost idling.

  • Mahfuz_SS_EHLMahfuz_SS_EHL Member, Provider

    @sibaper said:
    whats the bottleneck from observation?
    Do you already put cache somewhere?

    that server can handle the traffic, but without more information about your setup I only can assume that

    I saw MySQL going down multiple time, I increased the connections but that didn't add much improvement. The Server has NVMe but speed was around 600 Mbps (Don't know why). Server load was going upto 25-30 (E3-1275 v5), so it couldn't serve much. I assumed Network speed for a time being but couldn't be sure what was causing the issue. On the different note, the client has CloudFlare $20 Plan infront of everything.

    AlphaSSL Revocation Issue is being investigated.

  • The easiest way would be to add a layer of reverse proxy & cache (Nginx, ATS, Varnish, any of them would do) in front of your webserver. And of course, your server itself should have proper caching in the first place, including OPCache, Memcached/LSMCD/Redis, and LSCache.

    Thanked by 2Mahfuz_SS_EHL postcd

    One-click CDN server installation on your own VPS: OneClickCDN
    One-click browser-accessible Desktop environment on Linux VPS: OneClickDesktop

  • Whenever I see these, I’m always super curious who is serving this much traffic but can’t afford to pay someone to take care of it...

  • pbxpbx Member

    Full page cache + nginx or varnish... and a smaller machine will be able to handle the load easily. Unless there is a reason why you need this mysql activity that makes caching impossible?

    Thanked by 1Mahfuz_SS_EHL
  • HosterlabsHosterlabs Member, Provider
    edited September 29

    Load Balancing. It might seem you reached a point where horizontal scaling makes sense. You can combine load balancing with nginx, varnish, memcached, whatever you want to make it more effective. But having traffic like that I would suggest a cloud provider like aws, where you can create auto-scaling rules and add multiple virtual machines. So you have 20 vm's running at max traffic and let's say 5 running when there is very little traffic.

    At this point you should seriously consider that, from my point of view.

    Best regards!

    [Signature violates rules]

  • @iwaswrongonce said:
    Whenever I see these, I’m always super curious who is serving this much traffic but can’t afford to pay someone to take care of it...

    The same guys who love to maintain his own Ferrari.

  • whats the metric? show us your #top -c
    you should investigate from that.
    if your CPU high usage in mysq, then it must be bottleneck in some writing/insert query.

    look to your mysql. see what is running using command "show processlist"
    make index if possible, use cache,
    LS cache are possible to use redis or memcache. '
    just install at server, make sure its running and activate on lscache

    Thanked by 2Mahfuz_SS_EHL postcd
  • yokowasisyokowasis Member
    edited September 30

    is this a simple wordpress site ? e.g. a simple server content ? or a complicated wordpress sites ? e.g. ecommerce, LMS where people logged in and out ? and do stuff ?

    I have an old E5645, that can handle 16000 wordpress concurrent user easily. it stay at 30% cpu usage.

    Thanked by 1Mahfuz_SS_EHL
  • Mahfuz_SS_EHLMahfuz_SS_EHL Member, Provider

    @yokowasis said:
    is this a simple wordpress site ? e.g. a simple server content ? or a complicated wordpress sites ? e.g. ecommerce, LMS where people logged in and out ? and do stuff ?

    I have an old E5645, that can handle 16000 wordpress concurrent user easily. it stay at 30% cpu usage.

    WordPress Site, not simple, complicated a bit. They take exams of school students, they developed the exam system by ajax. The whole exam question is loaded at the beginning of exam but I assume their system saves every option the students picks in the question & writes to Database. That's why the so much database queries take place.

    AlphaSSL Revocation Issue is being investigated.

  • yokowasisyokowasis Member
    edited September 30

    @Mahfuz_SS_EHL said:

    @yokowasis said:
    is this a simple wordpress site ? e.g. a simple server content ? or a complicated wordpress sites ? e.g. ecommerce, LMS where people logged in and out ? and do stuff ?

    I have an old E5645, that can handle 16000 wordpress concurrent user easily. it stay at 30% cpu usage.

    WordPress Site, not simple, complicated a bit. They take exams of school students, they developed the exam system by ajax. The whole exam question is loaded at the beginning of exam but I assume their system saves every option the students picks in the question & writes to Database. That's why the so much database queries take place.

    PM me the website or a screenshot should be fine. I think they use my software.

    Thanked by 1Mahfuz_SS_EHL
  • Mahfuz_SS_EHLMahfuz_SS_EHL Member, Provider

    @yokowasis said:

    @Mahfuz_SS_EHL said:

    @yokowasis said:
    is this a simple wordpress site ? e.g. a simple server content ? or a complicated wordpress sites ? e.g. ecommerce, LMS where people logged in and out ? and do stuff ?

    I have an old E5645, that can handle 16000 wordpress concurrent user easily. it stay at 30% cpu usage.

    WordPress Site, not simple, complicated a bit. They take exams of school students, they developed the exam system by ajax. The whole exam question is loaded at the beginning of exam but I assume their system saves every option the students picks in the question & writes to Database. That's why the so much database queries take place.

    PM me the website or a screenshot should be fine. I think they use my software.

    Send you PM with SS.

    AlphaSSL Revocation Issue is being investigated.

  • Mahfuz_SS_EHLMahfuz_SS_EHL Member, Provider

    @dedipromo said:
    The easiest way would be to add a layer of reverse proxy & cache (Nginx, ATS, Varnish, any of them would do) in front of your webserver. And of course, your server itself should have proper caching in the first place, including OPCache, Memcached/LSMCD/Redis, and LSCache.

    Studied about LSMCD, seems it's a good idea to implement this. Can OPCache & LSMCD be used together ??

    AlphaSSL Revocation Issue is being investigated.

  • yokowasisyokowasis Member
    edited September 30

    @Mahfuz_SS_EHL said:

    @yokowasis said:

    @Mahfuz_SS_EHL said:

    @yokowasis said:
    is this a simple wordpress site ? e.g. a simple server content ? or a complicated wordpress sites ? e.g. ecommerce, LMS where people logged in and out ? and do stuff ?

    I have an old E5645, that can handle 16000 wordpress concurrent user easily. it stay at 30% cpu usage.

    WordPress Site, not simple, complicated a bit. They take exams of school students, they developed the exam system by ajax. The whole exam question is loaded at the beginning of exam but I assume their system saves every option the students picks in the question & writes to Database. That's why the so much database queries take place.

    PM me the website or a screenshot should be fine. I think they use my software.

    Send you PM with SS.

    It's not mine. From the looks of it, it use :

    https://wplms.io/

    or

    https://themeforest.net/item/wplms-learning-management-system/6780226

    As far as I know (because I am using it too), it doesnt save directly to the database everytime a student click an answer.

    It save to the database, when the student submit the test or when they click save the test button.

    But yes, that theme is heavy. It will install a butload of additional plugin. Ranging from buddypress to woocommerce.

    Also the database structure is just bad (for a test that is). It heavily use meta, and every question is count as 1 wordpress post. So if a test consists of 40 questions, it means 1 test need to load 40 wordpress post, multiply that to 800 user (because the nature of the test, everyone logged in at the same time). Because it uses user login, there is no way you can cache the page, unless, you modify the source code (this is what I did) and using partial page cache.

    I myself use this software for e-learning purposes, but not the exam / the test. I develop my own exam software to handle thousands of concurrent user and currently serving over 300 schools. It can be fully hosted on cdn, so it doesn't stress the server too much. If you offer services focused on education, and want to know more, feel free to pm me.

    Thanked by 2Mahfuz_SS_EHL Aidan
  • yokowasisyokowasis Member
    edited September 30

    @Mahfuz_SS_EHL said:

    @dedipromo said:
    The easiest way would be to add a layer of reverse proxy & cache (Nginx, ATS, Varnish, any of them would do) in front of your webserver. And of course, your server itself should have proper caching in the first place, including OPCache, Memcached/LSMCD/Redis, and LSCache.

    Studied about LSMCD, seems it's a good idea to implement this. Can OPCache & LSMCD be used together ??

    Tried this (nginx reverse proxy cache), full page cache, just doesn't work. The student's username on top right will just show the one who got cached. Even when the user logged as an admin. The only proper way to do this is modifying the source code, and implement partial page cache / query cache using varnish and the like.

    Thanked by 1Mahfuz_SS_EHL
  • @Mahfuz_SS_EHL said:

    @dedipromo said:
    The easiest way would be to add a layer of reverse proxy & cache (Nginx, ATS, Varnish, any of them would do) in front of your webserver. And of course, your server itself should have proper caching in the first place, including OPCache, Memcached/LSMCD/Redis, and LSCache.

    Studied about LSMCD, seems it's a good idea to implement this. Can OPCache & LSMCD be used together ??

    Yes, they cache different things and can be used together.

    Thanked by 2Mahfuz_SS_EHL Shamli

    One-click CDN server installation on your own VPS: OneClickCDN
    One-click browser-accessible Desktop environment on Linux VPS: OneClickDesktop

  • eva2000eva2000 Member
    edited September 30

    @yokowasis said: Tried this (nginx reverse proxy cache), full page cache, just doesn't work. The student's username on top right will just show the one who got cached. Even when the user logged as an admin. The only proper way to do this is modifying the source code, and implement partial page cache / query cache using varnish and the like.

    Yeah Wordpress guest caching is easy and logged in members is a lot harder. If you had >$5k/month you could try Cloudflare Enterprise which allows custom cache keys to fine tune what is cached at CF CDN level https://support.cloudflare.com/hc/en-us/articles/115004290387-Creating-Cache-Keys

    Here's what Cloudflare Custom Cache Key page rule looks like

    Thanked by 1Mahfuz_SS_EHL
    * Centmin Mod Project (HTTP/2 support + ngx_pagespeed + Nginx Lua + Vhost Stats)
    * Centmin Mod LEMP Stack Quick Install Guide
  • jsgjsg Member

    @Mahfuz_SS_EHL said:
    A website built on WordPress, concurrent hits/requests about 700-800 (shows in LiteSpeed WebAdmin), what is the best practice to serve it ?? I assume the following:

    Core: 8-10 Core Processor (Single Thread Performance should be good)
    Ram: Not less than 32 GB
    WebServer: LiteSpeed
    Disk: NVMe SSD
    MySQL Max Connection: 500-1000
    With increasing php_max_connection

    I want to know what you would do in this situation ? What kind of server, what kind of setup etc.

    Are you joking? Anything below 1000 req/s shouldn't be a problem at all and can be handled by 2 vCores or HTs; 4 vCores would be nice but luxurious.
    32 GB is quite likely more than what's needed, 4 GB - 8 GB seems to be about right incl. some reserve. NVMe is nice but SSD (preferably Raid 10) should do fine.

    I'd presume that your major problem is DB buffering and possibly a poorly designed web-application throwing lots and lots of "sub requests" within one http request at the server e.g. by pushing every user click to the server. Have a good look at your DB server's config, in particular about buffering (note that most DB servers are default optimized for high read ("SELECT") performance because that is what most people need).

    Thanked by 1Mahfuz_SS_EHL

    Thanks no.

  • Wordpress is pro in bogging down the resources. There was a case where simple corporate website was hogging the resources of a powerful server.
    Even using caching plugin, it was not working however, it was the theme.

    Thanked by 1Mahfuz_SS_EHL
  • Mahfuz_SS_EHLMahfuz_SS_EHL Member, Provider

    @jsg said: I'd presume that your major problem is DB buffering and possibly a poorly designed web-application throwing lots and lots of "sub requests" within one http request at the server e.g. by pushing every user click to the server. Have a good look at your DB server's config, in particular about buffering (note that most DB servers are default optimized for high read ("SELECT") performance because that is what most people need).

    Yes, I assume the same & that's why I've posted here to get some more idea.

    AlphaSSL Revocation Issue is being investigated.

  • jsgjsg Member

    @Mahfuz_SS_EHL said:
    Yes, I assume the same & that's why I've posted here to get some more idea.

    Here's your idea: look at the MariaDB documentation. There are (at least there should be) config vars like [something]write_buffer[something] which boil down to writing more into RAM (much faster) and less onto the disk.

    Btw, I doubt that not even clicking "Thanks" when people try to help you doesn't serve your interest ...

    Thanks no.

  • Look at mysql cache query. People have suggested me that on similar thread of mine. I just doesnt know how it works. I prefer redis. But if you dont want to tamper with the source code , mysql query cache is the best you could do.

    Thanked by 1Mahfuz_SS_EHL
  • Mahfuz_SS_EHLMahfuz_SS_EHL Member, Provider

    @yokowasis said:
    Look at mysql cache query. People have suggested me that on similar thread of mine. I just doesnt know how it works. I prefer redis. But if you dont want to tamper with the source code , mysql query cache is the best you could do.

    I configured LSMCD for now. Let's check how it goes.

    AlphaSSL Revocation Issue is being investigated.

  • A seperate Database server helps or split the database up to different tables to spread the load, can also try a MySQL Cluster.

    Thanked by 1Mahfuz_SS_EHL
  • nowthisisfunnowthisisfun Member
    edited October 2

    Did you try tuning the DB? It helped me a lot when running a Moodle server for ~2k concurrent users, had the same problem - DB server kept going down. https://docs.moodle.org/39/en/Performance_recommendations#Database_performance

    Thanked by 1Mahfuz_SS_EHL
  • FYI, Cloudflare just released an update to their Wordpress plugin for Automatic Platform Optimizations which uses CF workers to cache full HTML pages for guests https://blog.cloudflare.com/automatic-platform-optimizations-starting-with-wordpress/

    Might help only for guest side. But your member login side will need backend optimisations for cache miss requests.

    I did some benchmarks for CF Wordpress plugin's APO settings at https://community.centminmod.com/threads/cloudflare-wordpress-plugin-automatic-platform-optimization.20486/

    * Centmin Mod Project (HTTP/2 support + ngx_pagespeed + Nginx Lua + Vhost Stats)
    * Centmin Mod LEMP Stack Quick Install Guide
  • @Mahfuz_SS_EHL said:

    @sibaper said:
    whats the bottleneck from observation?
    Do you already put cache somewhere?

    that server can handle the traffic, but without more information about your setup I only can assume that

    I saw MySQL going down multiple time, I increased the connections but that didn't add much improvement. The Server has NVMe but speed was around 600 Mbps (Don't know why). Server load was going upto 25-30 (E3-1275 v5), so it couldn't serve much. I assumed Network speed for a time being but couldn't be sure what was causing the issue. On the different note, the client has CloudFlare $20 Plan infront of everything.

    You probably need to increase the maximum ram usage for MySQL by increasing the buffer or you could be going OOM resulting in the high loads.

  • @mezoology said:

    @iwaswrongonce said:
    Whenever I see these, I’m always super curious who is serving this much traffic but can’t afford to pay someone to take care of it...

    The same guys who love to maintain his own Ferrari.

    These guys, to the degree they actually exist, probably know what they're doing.

    Without any concrete data, this is just going to be a list of every possible thing you could do. OP doesn't even know what the bottleneck is...

    Thanked by 2Mahfuz_SS_EHL pbx
  • yokowasisyokowasis Member
    edited October 5

    @iwaswrongonce said:

    @mezoology said:

    @iwaswrongonce said:
    Whenever I see these, I’m always super curious who is serving this much traffic but can’t afford to pay someone to take care of it...

    The same guys who love to maintain his own Ferrari.

    These guys, to the degree they actually exist, probably know what they're doing.

    Without any concrete data, this is just going to be a list of every possible thing you could do. OP doesn't even know what the bottleneck is...

    in my experience handling students exam and using the same software, the bottleneck is the php use a lot of cpu, not to mention the way wordpress and the theme works. out of the box mysql is capable of handling thousands of concurrent connection let alone sub 1000. It's just php that is slow handling concurrent connection. that's why I migrate to nodejs.

    But I think, the exam is already over. exam usually last for a week. It's either OP has solved the problem, or the client using different exam solution (my guess is google form).

    Thanked by 1Mahfuz_SS_EHL
  • Jona4sJona4s Member
    edited October 5

    I get sick to my stomach whenever someone talks about scalability and mentions the words "Wordpress" and "PHP".

    Will humanity ever learn what a compiled language is, I wonder.

    let alone what a log-structured merge-tree / B-Tree with O(log N) "database" + a trillion locks, is doing to cause the high %sy and call it a "bottleneck"

    Thanked by 2jsg Mahfuz_SS_EHL
  • jsgjsg Member

    @Jona4s said:
    I get sick to my stomach whenever someone talks about scalability and mentions the words "Wordpress" and "PHP".

    Will humanity ever learn what a compiled language is, I wonder.

    Some reasonably good level of knowing and understanding databases before working with one would already go quite far, though.

    Thanked by 1Mahfuz_SS_EHL

    Thanks no.

  • Mahfuz_SS_EHLMahfuz_SS_EHL Member, Provider

    @iwaswrongonce said:

    Without any concrete data, this is just going to be a list of every possible thing you could do.

    There is no serious vulnerability in testing I think, if it's not at production level ;)

    AlphaSSL Revocation Issue is being investigated.

  • @Jona4s said:
    I get sick to my stomach whenever someone talks about scalability and mentions the words "Wordpress" and "PHP".

    Will humanity ever learn what a compiled language is, I wonder.

    let alone what a log-structured merge-tree / B-Tree with O(log N) "database" + a trillion locks, is doing to cause the high %sy and call it a "bottleneck"

    Any recommendation of high performance compiled language.

    Nodejs technically is a compiled language, anything better than it?

  • @yokowasis said:

    @Jona4s said:
    I get sick to my stomach whenever someone talks about scalability and mentions the words "Wordpress" and "PHP".

    Will humanity ever learn what a compiled language is, I wonder.

    let alone what a log-structured merge-tree / B-Tree with O(log N) "database" + a trillion locks, is doing to cause the high %sy and call it a "bottleneck"

    Any recommendation of high performance compiled language.

    Nodejs technically is a compiled language, anything better than it?

    golang, includes http server..one binary for everything

  • @yokowasis said:
    Nodejs technically is a compiled language, anything better than it?

    But it isn’t. Node isn’t a language. It’s a Javascript interpreter (which incidentally means it’s also not compiled).

    Thanked by 2jsg TimboJones
  • yokowasisyokowasis Member
    edited October 6

    @iwaswrongonce said:

    @yokowasis said:
    Nodejs technically is a compiled language, anything better than it?

    But it isn’t. Node isn’t a language. It’s a Javascript interpreter (which incidentally means it’s also not compiled).

    Well, you're nt wrong.

    I said nodejs. Because it serve a compiled javascript. So, technically it's a compiled language.

  • Mahfuz_SS_EHLMahfuz_SS_EHL Member, Provider

    @yokowasis An update, I installed LSMCD & OpCache & also configured LSCache, it decreased the server load a lot :smiley:

    AlphaSSL Revocation Issue is being investigated.

  • @Mahfuz_SS_EHL said:
    @yokowasis An update, I installed LSMCD & OpCache & also configured LSCache, it decreased the server load a lot :smiley:

    Great. Although for the server spec, it really is overkill. It shouldn't be that heavy in the first place.

  • Adam1Adam1 Member

    @Mahfuz_SS_EHL said:
    @yokowasis An update, I installed LSMCD & OpCache & also configured LSCache, it decreased the server load a lot :smiley:

    well done 👍

  • @Mahfuz_SS_EHL said:
    @yokowasis An update, I installed LSMCD & OpCache & also configured LSCache, it decreased the server load a lot :smiley:

    Is it still the same server? What's your typical CPU usage now?

  • Mahfuz_SS_EHLMahfuz_SS_EHL Member, Provider

    @agentwp said:

    @Mahfuz_SS_EHL said:
    @yokowasis An update, I installed LSMCD & OpCache & also configured LSCache, it decreased the server load a lot :smiley:

    Is it still the same server? What's your typical CPU usage now?

    No previously, it was a hetzner server with cPanel (E3-1275 v5 with NVMe). Now, it's php-friends 8 Core Plan with 40 GB Ram with DirectAdmin, I forgot the NVMe Disk Space.

    AlphaSSL Revocation Issue is being investigated.

  • @iwaswrongonce said:
    Whenever I see these, I’m always super curious who is serving this much traffic but can’t afford to pay someone to take care of it...

    I'm guessing the OP is the hired maintainer but trying to figure it hence posting it here. Just my wild assumption.

Sign In or Register to comment.