Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Web server advice
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Web server advice

DeorDeor Member
edited September 2012 in Help

Guys, got a bit of a situation here and im looking for some input, short term and longer term.

Currently have a VM that we use for basic hosting after the physical machine we had used for years started to fail. Its a VMware instance with 512Mb of ram on a pretty high end blade system. Running Debian 6, Apache, PHP MySQL, ProFTPd, Webmin and Exim4.

There are about 50 domains, mostly low traffic, a handful of databases and one SSL site.

The problem is that the server keeps grinding to a halt and becoming unresponsive. The only options at this point is to force a power off from the VMware client and reboot it. From what the host is saying, there just isnt enough RAM for the system, which i find hard to believe, and when things like crawlers are hitting our sites its not able to cope.

Why is apache not just timing out, rather then trashing the whole system?

I did follow some config guides when i configured the server, but what would you guys recommend for configuring apache, php, mysql etc to get some stability right now?

I know switching over to Nginx will probably help no end, so i need to look at that i think. How do you migrate from an an apache config file with virtual hosts, over to nginx anyway?

For the record, this system has been running since april and was very stable to start with, but we have been having more and more problems recently and i just dont know what im looking for to sort it.

Would really appreciate any input as its giving me a real headache.

Comments

  • I'd suggest a 2 pronged approach:
    1. Either add more virtual RAM and more swap (can you post output of swapons -s) to provide some temporary relief
    2. Enable process accounting (accton) to identity programs using the most resources or implement munin/nagios/whatever so you can take a more data driven approach to figuring out where to focus your energy. The monitoring will add some load of course but in my experience it's 5-10% max.

    I don't recommend switching web server software unless you are more familiar with it.

  • I know switching over to Nginx will probably help no end, so i need to look at that i think. How do you migrate from an an apache config file with virtual hosts, over to nginx anyway?

    Mostly manual conversion, I'm not aware of anything automated so far.

    However, it's extremely great once you do make the initial jump. It felt like a pain when I jumped ship in 2010, but ever since, I've had no regrets.

    If you're not familiar with nginx, I'd recommend hiring a sysadmin to lend you a hand.

    I'm available if you'd like to talk :)

    </self advertising>

  • Thanks @craigb the host recommended taking the VM up to 1gb of ram, but that bumps the cost up to £50 a month :/

    There is 2Gb of swap by the look of it

    Filename                                Type            Size    Used    Priority
    /dev/sda5                               partition       1951736 52444   -1
    
                 total       used       free     shared    buffers     cached
    Mem:        508772     415628      93144          0      15756     247800
    -/+ buffers/cache:     152072     356700
    Swap:      1951736      52204    1899532
    

    Will have a look at accton i think!

    @Wintereise thanks for that, i'll keep you in mind! The DC has good techs thats ive known for years, but they aint cheep!

  • @deor ok so you have swap configured - that's good. Any errors in your system, apache, MySQL logs?

  • 50 domains on 512mb. Wow

  • @cosmicgate said: 50 domains on 512mb. Wow

    Can you expand on that a little @cosmicgate? is that a good wow or a bad wow?

  • Is there any reason for the sites to run on a "high end blade system" instead of on a LEB with a reputable hoster where 1gb RAM costs like 10€/month?

    Thanked by 2nocom ChrisPV
  • MaouniqueMaounique Host Rep, Veteran

    With SSD and all...
    You can get a dedicated server for 50 pounds a month, and wont be just the regular atom...
    M

  • History really @gsrdgrdghd. The DC is about a 35minute drive from our office and we originally had 2 physical servers racked there starting in early 2007. A windows mail server and a linux web server with a sonicwall in front of them.

    When we had hardware issues with the windows box, we migrated it to their VMware cloud and its been rock solid since. When the time came to do the same with the old linux server we did the same thing. If i was doing the same now, with what i know about the LEB world, i would have put up with 24 hours of DNS propagation and saved a lot of money!

  • @Deor said: I did follow some config guides when i configured the server, but what would you guys recommend for configuring apache, php, mysql etc to get some stability right now?

    install munin
    reconfigure Apache(again) drop inodb from Mysql and reconf mysql(again), do you relly need ProFTPd? also you can implement some cash system xcash etc.
    Take a look of Munin graphs

  • serverbearserverbear Member
    edited September 2012

    £50 for 1GB is pretty expensive (around these parts).

    Heck you can get an i5 @ Kimsufi with 16GB RAM for £35.99/month.

    My 2c, start with a new server, move to Nginx (run it in parallel with the existing server until you figure out all your Rewrites etc), use Percona build of MYSQL. If you can get away without a control panel you'll be flying :)

    Another option is running a benchmark & comparing to some of the offers around here to see how you actually stack up performance wise (I know you say a "high end blade system" but what exactly are the specs?).

    Thanked by 1craigb
  • jhjh Member
    edited September 2012

    @serverbear said: £50 for 1GB is pretty expensive (around these parts).

    There may be factors besides spec that justify it. I pay £50/month for 512MB RAM because the company, hardware and network's stability and quality justify it, and those things are more important than spec to me :P

  •  total       used       free     shared    buffers     cached
    Mem:        508772     415628      93144          0      15756     247800
    -/+ buffers/cache:     152072     356700
    Swap:      1951736      52204    1899532

    Am I the only one who doesn't see why he needs a gb of ram? Running 50 low traffic domains on 512 ram isn't too much of an issue if you're not using a heavy control panel. If this result is accurate then there is plenty of ram free, sure 1gb would be lovely but I don't see why you need it. Optimizing mysql is a nice idea but I'd look into the logs to find out exactly whats crashing and why.

  • Was curious about the Percona build of MySQL mentioned by @serverbear. Stumbled across this well written HOWTO (switch) in the process for CentOS: http://docs.1h.com/Percona_MySQL_installation_tutorial

  • I would create a . all.log file via rsyslog.conf in an attempt to try to track the cause of the server crashing.. It may not necissarily be memory related

  • @Deor even if the company justifies the usage of a high price there still other companies that have better prices and better support, take a look at linode 39.95 USD gets you 1GB RAM, 40GB HD and 400GB bandwith.

    Its a matter of shop around and search for respectable providers, you can also try a LEB box, Prometeus, Hostigation, BuyVM, SecureDragon are respectable providers here, try their location and services.

    If you want to have best uptime possible, go for an outside DNS (zonedit, he.net) and have 2 LEB's sincronize with all your data, in the end its pretty sure that you will spend less or the same amount of money and you will increase your uptime.

  • Thanks for the feedback all, very useful stuff. In the end i got to the bottom of the immediate problem which was down to one customer having set up forum with no registration checking. It had been found by bots that were hitting it with new user registrations and spam forum posts in huge numbers. I deleted the forum and set it up properly and things have been a lot quieter!

    All of this has, however, made us think about exactly what we are getting for our money and what we really need. I will be looking to move elsewhere once the contract is up for renewal in 6 months time!

  • MaouniqueMaounique Host Rep, Veteran

    @Deor said: All of this has, however, made us think about exactly what we are getting for our money and what we really need

    That is a question that should be running loops all the time :) What do I need, how can I get it, how much will cost in money and time to setup, how scalable is it, how pleasant the experience.
    M

  • Very true @Maounique, very true :)

  • @Deor I had a similiar issue and cloudflare stopped 95 percent of the bots submitting forms. Also used the geoip module in nginx to deny a certain country. But you're right, there should be checks for legitimate accounts, emails, and/or capatcha

  • In the days when everyone can buy gobs of cheap VPSes all around the world blocking by country does not seem like a very effective way to protect yourself.

  • innyainnya Member
    edited October 2012

    checked your cached it has 247800.

    I also encountered the simlar problem with VMWARe that did a lot of cache.
    I just schedule crone job every 10 to 20 minutes to release the cache to work around the problem.

    Also, use ISAM instead of INNODB in Mysql.

  • bdtechbdtech Member
    edited October 2012

    @rds100 correct, just sharing what worked for me. If a huge majority of spam and bots is coming from one country that isn't a key region for your site you can throw capatchas up first based on country and redirect or assign a variable if needed. This ensures you dont hassle all your users, just those of higher risk.

    Cloudflare actually uses similar tactics to stop spam bots

  • Maybe someone should come up with a spamBL / dnsBL but for spam bots that you can drop into CSF. I've blocked Kimsufi and Relakks (VPN service that doesn't care), with spam dropping down pretty well.

  • risharderisharde Patron Provider, Veteran

    I've experienced something like this before on a VMWARE running on a blade and what I did was tweaked my mysql settings since it was mysql that was using up the most ram. For some reason, once the RAM was nearly used up, it would always freeze even though I allocated a whopping 5GB swap. Though this is not a full solution for you, I hope it helps you in some way. I suspect there's something up with how VMWARE manages RAM or the virtualized hardware that the OS detects simply cannot do what its supposed to do which is to begin using up the swap instead of halting/freezing.

Sign In or Register to comment.