Website keeps running extremely slow

hostnoob

I have a 512MB VPS running lighttpd, PHP and MySQL. It runs IP.Board forum, and uses around 130mb when it's quiet, to 250-300 when it's busy.

It's been fine for like 2 years, but suddenly it starts hanging randomly. It will be fine, then suddenly pages won't load when you refresh. Then a few minutes later it will load fine, then maybe 30 seconds later it starts hanging again.

It's happening right now, and using 131MB RAM, and the load average is load average: 0.51, 0.55, 0.48 (Intel(R) Xeon(R) CPU E5649 @ 2.53GHz * 24 cores)

It seems to be fine whenever I restart the lighttpd service. I've messed around with the lighttpd config and how many children processes fastcgi spawns, etc.. but it keeps happening. I'm not sure what logs I can use to troubleshoot it.

It's also downloading at 47MB/s and I can download from the server at my home connection's max speed when I test that so I don't think it's a bandwidth issue

neroux

Any errors in the logs?

Floris

Mind telling me the output of iostat 1

zionvps

check all logs in var/log. 90% of errors are reported there
do a 1 min ping test to check packet loss
try disabling certain lighttpd modules of lighttpd one after another, like rewrite, fastcgi, auth etc

hostnoob

@neroux said:
Any errors in the logs?

Not from when it started happening, although yesterday I changed fcgi max requests to 100 (thinking when it hit that it would reset the process and it would be okay without me having to restart it) so I have these errors

2014-04-20 10:27:28: (server.c.1444) [note] sockets disabled, connection limit reached
2014-04-20 10:28:17: (server.c.1398) [note] sockets enabled again

@Floris said:
Mind telling me the output of iostat 1

iostat 1
Cannot find disk data

Floris

How about:

sudo iotop

hostnoob

@Floris said:
How about:

sudo iotop

iotop
Could not run iotop as some of the requirements are not met:

• Linux >= 2.6.20 with
  • VM event counters (CONFIG_VM_EVENT_COUNTERS)

it may be because it's vserver not openvz or xen or anything (2.6.32-5-vserver-amd64)

black

Have you tried switching to a different web server like nginx? Swap out lighttpd for nginx and see if that's fixed. If it's not, then you know it's not a problem with lighttpd. Might want to do this with php as well (with php5-fpm).

rmlhhd

I didn't know a website had legs?

user123

@rmlhhd said:
I didn't know a website had legs?

Googling shows that there are many websites with legs, including (NSFW) hotlegsandfeet.com (NSFW).

hdpixel

I would check the current connections to the server. Often, there are bots scanning the site and making tons of unnecessary connections. I always have the mysqltuner.pl handy and run it at the worst time.

show tcp connections
netstat -anp -tcp

established connections
netstat -tn

mysql> SHOW STATUS WHERE `variable_name` = 'Threads_connected’;

mysql>SHOW STATUS WHERE `variable_name` = 'Threads_running';

black

@user123 said:
Googling shows that there are many websites with legs, including (NSFW) hotlegsandfeet.com (NSFW).

I went there just because it was tagged as NSFW. Gotta say, the easter bunny is kinda creepy.

hostnoob

Still ongoing, this is driving me crazy now.

@black said:
Have you tried switching to a different web server like nginx? Swap out lighttpd for nginx and see if that's fixed. If it's not, then you know it's not a problem with lighttpd. Might want to do this with php as well (with php5-fpm).

I've tried nginx but I always liked how lighttpd was easy to set up with PHP and performed well, plus I know the settings for it in the config file.

maybe I could set nginx up to run on port 8x, configure it and then stop lighttpd and change nginx to port 80. hmm.. do you have any guides on configuring nginx with PHP (and setting up vhosts)?

@hdpixel said:
I would check the current connections to the server. Often, there are bots scanning the site and making tons of unnecessary connections. I always have the mysqltuner.pl handy and run it at the worst time.

> show tcp connections
> netstat -anp -tcp
> 
> established connections
> netstat -tn
> 

mysql> SHOW STATUS WHERE `variable_name` = 'Threads_connected’;
> 
> mysql>SHOW STATUS WHERE `variable_name` = 'Threads_running';
> 

What sort of things should I be looking out for in netstat?

AThomasHowe

Nginx and lightttpd are both I'll suited to high traffic dynamic sites like with PHP. Apache actually performs a lot better. It's a learning process but the best way to go would be to set up nginx on port 80 to serve static content and back proxy PHP and other dynamic content back through apache on another port (8080 for example).

It sounds scary but if you worked out lightttpd I think you could get it going in a few hours. Maybe buy a cheap box and try setting up on a fresh environment or another box from the same provider you can move over onto.

I am on mobile because I'm off to bed soon but if you don't get some good links I'll post some tomorrow.

hostnoob

@AThomasHowe said:
Nginx and lightttpd are both I'll suited to high traffic dynamic sites like with PHP. Apache actually performs a lot better. It's a learning process but the best way to go would be to set up nginx on port 80 to serve static content and back proxy PHP and other dynamic content back through apache on another port (8080 for example).

It sounds scary but if you worked out lightttpd I think you could get it going in a few hours. Maybe buy a cheap box and try setting up on a fresh environment or another box from the same provider you can move over onto.

I am on mobile because I'm off to bed soon but if you don't get some good links I'll post some tomorrow.

Thanks for the reply, it was actually working fine until about a week ago. Always used about 380mb RAM max on a 512 box (including MySQL) and never crashed. There hasn't been an increase in posting either.

Anyway, I've just installed nginx and php5-fpm. I've left the default settings, but changed these

server_name localhost; to server_name _;

added index.php to index

and here's the PHP bit (I changed cgi.fix_pathinfo to 0 in php.ini)

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    location ~ \.php$ {
            fastcgi_split_path_info ^(.+\.php)(/.+)$;
    #       # NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini
    #
    #       # With php5-cgi alone:
    #       fastcgi_pass 127.0.0.1:9000;
    #       # With php5-fpm:
            fastcgi_pass unix:/var/run/php5-fpm.sock;
            fastcgi_index index.php;
            include fastcgi_params;
    }

Are there any security issues I need to fix?

AThomasHowe

not really, but what you should do is make a new ghost file (in /etc/nginx/sites-enabled, it can be called whatever you want) and it's good practice to listen on www.your.domain and your.domain in server_name, maybe .server.name (dot at the beginning) to catch all subdomains.

You could also set it up so all HTTP traffic is automatically forwarded to HTTP, then put all your server configs under the ssl bit.

There's some stuff you could add for speed and performance, cache control for static content etc but for real I am off to bed now, so I will post some links if needed in the morning. Good luck!

I think in the long run you might find it useful to try nginx + apache though, for future projects and future growth. Up to you though. You could also use cloud flare to speed up your site and cache static content. If your issue is php through fastcgi php-fpm may help a little but in the end it's not going to make a huge difference once it gets to processing php between nginx and lighttpd. It may work for now though.

hostnoob

@AThomasHowe said:
not really, but what you should do is make a new ghost file (in /etc/nginx/sites-enabled, it can be called whatever you want) and it's good practice to listen on www.your.domain and your.domain in server_name, maybe .server.name (dot at the beginning) to catch all subdomains.

You could also set it up so all HTTP traffic is automatically forwarded to HTTP, then put all your server configs under the ssl bit.

There's some stuff you could add for speed and performance, cache control for static content etc but for real I am off to bed now, so I will post some links if needed in the morning. Good luck!

I think in the long run you might find it useful to try nginx + apache though, for future projects and future growth. Up to you though. You could also use cloud flare to speed up your site and cache static content. If your issue is php through fastcgi php-fpm may help a little but in the end it's not going to make a huge difference once it gets to processing php between nginx and lighttpd. It may work for now though.

Okay, thanks for your help. Off to a bad start though since nginx started giving 502 bad gateway errors about 30 minutes later (was running fine before that)