Help me with Apache2.4 with MPM Event and PHP-FPM

warrior

Hi, my webserver look bottleneck. its fast at first, load website only in 1s but after few hours its turn to timeout (more than 60s). my server is: E3-1240 v3 with 32Gb Ram (Debian 9). and current usage its only 6Gb, previously im use MPM Prefork and eat 20Gb RAM, and i switch to MPM Event, same as prefork problem, but bottleneck timeout take a days.

my current config is:

MPM Event

<IfModule mpm_event_module> StartServers 8 MaxClients 256 MinSpareThreads 64 MaxSpareThreads 256 ThreadLimit 64 ThreadsPerChild 25 MaxRequestWorkers 12800 MaxConnectionsPerChild 0 </IfModule>

PHP-FPM Pool

listen = 127.0.0.1:9000 pm = dynamic pm.max_children = 64 pm.start_servers = 8 pm.min_spare_servers = 4 pm.max_spare_servers = 12

vHost

<FilesMatch "\.php$"> SetHandler "proxy:fcgi://127.0.0.1:9000/" </FilesMatch>

i like to ask at serverfault/stackoverflow but i think peoples here have more experince cause its specific forum. please help what my problem with my config, im just host larave and wordpress only sites that currently i host with 150s domain. thanks

warrior

Update : im confuse with apache2buddy test result:

[ -- ] Your server has 32138 MB of PHYSICAL memory.
[ >> ] ServerLimit directive not found, assuming default values.
[ -- ] Your ServerLimit setting is 16.
[ -- ] Your ThreadsPerChild setting is 25.
[ -- ] Your ThreadsPerChild setting for worker MPM is  25
[ -- ] Your ServerLimit setting for worker MPM is  16
[ -- ] Your MaxRequestWorkers setting is 12800.
[ >> ] MaxClients directive is higher than ServerLimit, using ServerLimit (16) to apply calculations.
[ @@ ] Current Apache vHost Count is greater than maxrequestworkers, which is unusual, but can be valid in some scenarios.
[ >> ] MaxClients has been hit recently (maximum of 5 results shown), consider the dates and times below:
[Sat Mar 31 08:15:23.492484 2018] [mpm_event:error] [pid 25884:tid 140011331273920] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[Sat Mar 31 08:27:11.200190 2018] [mpm_event:error] [pid 27989:tid 140623982032064] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[ @@ ] The following recommendations may be misleading - apache has been restarted within the last 24 hours.

Your server's physical RAM:                                   32138 MB
Remaining Memory after other services considered:             31746 MB
Apache's MaxRequestWorkers directive:                         16       <--------- Current Setting    
Apache MPM Model:                                             event
Largest Apache process (by memory):                           308 MB
Apache appears to be running in event mode.
Please check manually for backend processes such as PHP-FPM and pm.max_children.
Apache2buddy does not calculate maxclients for worker model

2018/03/31 17:22:02 Model: "Event" Memory: "32138 MB" Maxclients: "16" Recommended: "N\A" Smallest: "70.84 MB" Avg: "293.78 MB" Largest: "308.40 MB"

its i use MPM Event should i use MPM Worker to edit config, why its show MaxRequestWorkers 16.

warrior

please help!

i though its not apache, its php-fpm pool config. how to set the right pool config, any reference?

Falzo

if running into problems only after some time, it sounds like your processes get stuck over time blocking further requests. check your timeout settings and lower them to make sure stuck processes get ended properly. turn off keepalive and persistent connections (if in use) to see if that helps.

try to find the real cause for processes getting stuck in the first place - that most likely isn't an apache issue nor related to the php settings but a problem with the code from whatever you are using with your page.
check the error-logs, turn on more verbose php logging.

put some monitoring in place (f.i. monitorix) to be able to see how usage of RAM and number of processes evolve over time and how they might relate.

warrior

@Falzo said:
if running into problems only after some time, it sounds like your processes get stuck over time blocking further requests. check your timeout settings and lower them to make sure stuck processes get ended properly. turn off keepalive and persistent connections (if in use) to see if that helps.

try to find the real cause for processes getting stuck in the first place - that most likely isn't an apache issue nor related to the php settings but a problem with the code from whatever you are using with your page.
check the error-logs, turn on more verbose php logging.

put some monitoring in place (f.i. monitorix) to be able to see how usage of RAM and number of processes evolve over time and how they might relate.

hi thanks you for your valuable suggestion, finally i can find the problem. its php-fpm pool config, thats by default not set and use 0 (unlimited) value for pm.max_requests, docs bit of lack explanation for value calculation i think, and i try monitorix, that tools really help. thanks

FHR

I use pm.max_requests = 75. Seems to work fine.

TommySRV

Check also pm.max_children and pm.process_idle_timeout

warrior

@FHR said:
I use pm.max_requests = 75. Seems to work fine.

is it good to set it very low? i set it to 2000 currently.

@TommySRV said:
Check also pm.max_children and pm.process_idle_timeout

thanks for suggestion, currently use static and for pm.max_children = 128 i set by count average of php.

nah for pm.max_requests i dont find any referance how to set it, i set to 5000 after a day it become 503 status code, currently i set it to 2000 and seem fine, just wait a day or more.

how exactly to set pm.max_requests, any reference for this. thanks for help

FHR

pm.max_requests = 128 - a single PHP process will serve 128 requests before restarting itself. The value does not matter too much, it's to prevent memory leaks from having an impact. I use 75 on my servers but as I said, it doesn't matter too much.

I would also use pm = dynamic instead of static. It does not make much sense to use static.

warrior

its same dynamic or static, i use all resources with static config. the only problem is some time after few hours it show 503 Service Unavailable since i dont see anything at /var/log/php7.1-fpm.log

Zerpy

@FHR said:
it's to prevent memory leaks from having an impact. I use 75 on my servers but as I said, it doesn't matter too much.

It prevents memory leaks, sure - however 75 actually does matter in a high traffic environment at least.

The issue of shutting down a fpm process (same goes for fastcgi or any other handler that stays alive), is that when you're shutting it down and you do another request, the time it takes to start up the process again can be 50-150 milliseconds alone for the master process to start up a child and make it ready to do actual processing of your request.

Now, sure - 50-150 ms isn't a lot, but let's say you do 150 requests per second to PHP, this means in the span of 1 second, you'd have to shutdown and spawn two new processes, meaning you'll easy spend 10-20% of the time just spawning new processes - you've thus lost 10-20% of your capacity rather easily to useless connection handling.

You shouldn't set the thing to unlimited, but setting it to 2k or 10k in most environments doesn't matter much (things rarely leak so much within that request count and if it does, you have bigger issues).

At same time, some handlers (not php-fpm specific, since it's from the master process), you'll end up killing your shared memory cache (such as opcache), again affecting the performance.

Does it matter for small traffic sites? Probably not - but it surely does for sites with decent traffic. You could also "fix it" by using more hardware, but.. why give up capacity to silly configs.

warrior

@zerpy i think my server resource still can handdle it here, that why i try with different config and use ab test and still not find the problem that show 503 status code after few hours.

my server for this config is E3-1240 v3/32Gb/2Tb/1Gbps and when show 503 status code the load cpu (1.0), ram 12gb, and other is just fine. but not showing anything at /var/log/php7.1-fpm.log.

what usually bottleneck with 503 status code, i try with ab -n 5000 -c 100 http://mytestdomain.com/ and can run in 8s, but after few hour its become 503.

Abdussamad

Increase maxclients in apache config and pm.max_children in php fpm config to 384. See if that fixes the problem. Right now you are limiting yourself artificially with your unduly low settings and not allowing enough people to connect to your server.

warrior

@Abdussamad said:
Increase maxclients in apache config and pm.max_children in php fpm config to 384. See if that fixes the problem. Right now you are limiting yourself artificially with your unduly low settings and not allowing enough people to connect to your server.

thanks for suggestion, currently i set it to 1024 and 256.

warrior

i just realize that nothing wrong with my config, just see apache log that middle asian ips try to do something my server.

Abdussamad

1024 may be too high and 256 too low. that's why i suggested 384 and then raising it gradually over time until you arrive at an optimal number for your server. if you set it too high you run out of RAM and the kernel starts killing processes. if you set it too low you run out of workers to services incoming requests.

as for the bots probing your server that's normal. if they are wasting too much CPU time you can look at blocking them.

warrior

yeah look like so, but 256 should enough for now. i mean 1024 is for maxclients in apache.