All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Help me optimize my VPS
Hello LET!
I have a Xen VPS which has DirectAdmin running on a 32bit version of CentOS. It now has 1 GB of dedicated RAM and 1 GB swap. Previously, it had 512 MB. I upgraded few weeks back.
The problem I am having (and this is the reason I had upgraded from 512 MB to 1 GB in the first place) is that, the VPS works fine for a few days and then it comes down to it's knees and stops responding and needs to be rebooted. On boot the system starts at around 170 MB of RAM usage and then it swims between 250 and 500 MB. Now this is what happened.
Yesterday after being up for over 20 days I noticed my sites were not opening so I SSHed into the system to check what's going on. The system had exhausted all of it's dedicated RAM and was using the swap memory. Upon doing a ps aux I saw there were around 110 httpd entries and a lot of mysqld entries too. I don't remember exactly how many mysqld entries there were but there were a lot compared to the usual number. I believe this is referred to as 'thrashing'? Anyways, what I find weird is that, upon doing a reboot everything becomes normal. If there is a spike in traffic, the same thing should happen after reboot also, right? But it never does. It plays nice after the reboot.
I am completely lost here and would appreciate if somebody could tell me why this could be happening and point me to the right direction. I want to know if I should be looking into optimizing Apache, MySQL or PHP?
Thanks!
PS: The server powers around 16 domains, of which most get hardly any traffic. There are only 4-5 domains which get traffic.
Here are the combined daily stats from Webalizer:
Hits: 33745
Files: 29709
Pages: 15546
Visits: 5288
Comments
It will be difficult to optimize your VPS if you're using a control panel, or otherwise packaged software.
Regardless, it sounds like one of your domains is getting quite a bit of traffic in bursts, and this is causing your VPS to spawn many httpd processes. And then they may not be exiting after a time-out period.
There are directives you can set in Apache that will allow processes to exit more often, when they're not being used. Are you using the prefork or the worker MPM for apache?
For what I am reading, seems yes, he is using mpm, the default.
Google about the MaxSpareThreads, MaxClients and other variables
Here the spec
http://httpd.apache.org/docs/2.0/es/mod/mpm_common.html
Now look for reducing this variables.
My second advice, is to install PHP APC. Look for installing it on your system, it will increase the throughput of your machines and will reduce RAM usage drastically because PHP won't be processing all the cached stuff.
Third advice, look if you need InnoDB in your databases, dropping that engine and going for MyIsam 'can' reduce your ram usage (it depends, because is a Xen one).
Finally
Extreme advice 1: Drop your panel and Apache and use Nginx =P
Extreme advice 2: Drop Centos and use Debian n_n
Why?
My advice n_n (Me in his shoes).
Yes, but how would he benefit from this?
I'm assuming you're saying here that Debian is better than CentOS in his case.
I want to know why you say that, assuming it's not because you're used to it.
Was only saying because yum as a memory eater won't help if he frequently updates. But thinking a little bit, in Xen that may not cause these adverse effects.
And yes, about Debian I am used to it =P (uber fanboy here)
@Damian4IPXcore
I haven't modified the default apache configuration. It's using prework MPM with the following values:
StartServers 5
MinSpareServers 5
MaxSpareServers 10
ServerLimit 450
MaxClients 450
MaxRequestsPerChild 10000
Quoting:
Regardless, it sounds like one of your domains is getting quite a bit of traffic in bursts, and this is causing your VPS to spawn many httpd processes. "And then they may not be exiting after a time-out period."
I believe it's the part which I have wrapped in double quotes! I have noticed this many times in the past. None of my sites receive consistent traffic. So if there is a spike, regardless of it's frequency, the usage should drop back soon because the duration will be short. Yesterday it stayed at 1 GB for a whole 10 minutes and that's just impossible.
What is it that could be stopping them from terminating once they are done?
450 clients = 450 threads...
Sounds freaking insane, reduce that
MaxClients 450
too high even for a bigger server... Try reducing these value. Do you see kernel OOM errors?
Are you married to Apache? Lighttpd and nginx scale better, especially with concurrent connections. Are you serving static pages? Nginx or even Cherokee would be perfect and would significantly reduce your memory footprint. Apache is obviously a great web server, but in my experience it requires more deployment-specific tuning than other servers.
For dynamic content I'd agree with Yomero about installing APC. It's going to be included with php anyway starting with 5.4.
@Soylent: I don't think DirectAdmin works with either of those web servers but I am definitely going to try APC.
@yomero && @prometues: Thank you for pointing that out! At least now I know what I should be looking into to. I have set ServerLimit and MaxClients to 15. Do you think that's a good value for a 1 GB VPS? And yes, I have seen out of memory errors. It usually appears in the log when the server experiences the thrashing.
I think with 30-50 you will be ok... BUT, you need to try it, this depends on your websites.
Are you using mod_php?
cloudflare.
@prometues: No, it's running as FastCGI. I am hoping I am good here at least!
One point to you :-)
Did you disable keepalive in httpd.conf? At the cost of some reconnection overhead it should help keeping low the apache processes.
You can use nginx as reverse proxy to apache to see how things gone. Its easy to do this and you will reduce instandlly cpu and memory usage.
Googleit and you will find many tutorials.
@prometeus: I have done that just now. So far this is what I have done since yesterday:
Set KeepAlive to Off
Changed ServerLimit and MaxClients to 30
I didn't have a /etc/my.cnf so MySQL was on default. Yesterday I copied the following to /etc/my.cnf: http://fts.ifac.cnr.it/cgi-bin/dwww/usr/share/doc/mysql-server-5.0/examples/my-medium.cnf.gz
Should I use http://fts.ifac.cnr.it/cgi-bin/dwww/usr/share/doc/mysql-server-5.0/examples/my-small.cnf instead?
The VPS has been online from the past 14 hours. It was using around 400 MB of RAM even till 3 hours back but it's now sitting at 710 MB. I know this will keep rising and the system will thrash again.
Set a cron job that at 5:00 stop and start apache so that if it survive all day you are set for now.
Than I think you should look at replacing apache as suggested if you can...
His problem there is that he's running DirectAdmin? I did some google searches though, and it does look like people were able to get DirectAdmin running with lighttpd, but there's no auto install. I even saw a DirectAdmin setup with nginx running static pages and reverse proxying requests for dynamic pages to another server on another port.
So it's possible, but I couldn't tell you how to do it, or how hard it would be.
I tried many things but I still haven't found a solution for this issue. This is happening more frequently now. As a matter of fact, today is just exhausted all of it's memory and thrashed within 24 hours. Yesterday I had tried installing APC. I got it installed but for some reason, it wasn't working even when I hooked WordPress to use it. The memory usage stayed the same, 320KB of 128 MB I had assigned to it.
Anyways, I had a question pertaining to the amount of RAM that's available. Considering the situation, could it help if I upgraded my plan from 1 GB to 1.5 GB?
I have a directadmin box running quite well less than 1GB RAM. it never spiked more even though i have 2GB burst.
On boot it takes around 430MB RAM with apache, mysql with innodb
Disabling innodb gives back 100+Mb thus around 350RM
It runs a forum (phpbb) and a few wordpress blogs and also acts as VPN
Traffic ~350 unique visitors per day ( peak RAM usage 880MB)
Thinking of running litespeed but a bit lazy to configure DA
@peppr: Did you tweak your installation or are you on the default configuration? And which distro are you using?
Cent OS 5.7 32bit minimal install. DA custom build 1.2 with innodb disabled. i dont remember doing much to reduce ram usage. Custom build 2.0 released on 13th Feb. Planning to update once I manage the backups.
Set a cronjob to restart Apache once every 15 minutes. If you go 24 hours without maxing out your RAM, you then make small changes to your httpd-default.conf file and change your cronjob to once an hour. If you go another 24 hours without maxing out your RAM, remove the cronjob.
For basic settings try something like:
StartServers 5MinSpareServers 5MaxSpareServers 10ServerLimit 50MaxClients 50MaxRequestsPerChild 2000Also lower your timeout, turn off keepalive, remove modules you don't use, etc... If nothing helps, buy more RAM.
Ok, the problem happened again today. When I ssh'd in, the memory usage was consistent 820 MB (i.e. I didn't see any significant rise or fall). So I thought I would one by one stop all services and kill all processes just to make sure it's apache that's causing the problem. I thought it could be something else but to my surprise, I stopped all services and killed all the processes which I could and the memory usage came down only to 770 MB!
You can see the ps output here: http://pastebin.com/shM48Btb
Free: http://pastebin.com/axsDKfKf
The processes in the square brackets, I couldn't kill them even with a kill -9.
So now I am completely puzzled as to what the problem could be and what is it that's eating all my memory?
You need to look at the memory cache. Only 96MB was actually being used.
Linux by default use all the available memory for caching so the fact that you see the memory usage grow isn't bad. The problem is when you start swapping...
@naif lots of tweaks for mysql, disable unused apache modules, configure firewall ...
you can also limit your number of servers starting there ... do you run an mta ...?
what about some top output (top -s), do you use bind ... ? need a cpanel ... ?