New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Comments
Something's invoking the OOM killer due to memory usage, MySQL gets targeted because it has the highest score (presumably because it's the single largest process) at the time. The value you are seeing might be after the kill occurred, or there may be something trying to allocate large amounts of memory, exhausting it, and failing (eg: MySQL buffers/caches or spurts of large number of httpd/php-fpm processes).
What does your MySQL configuration look like?
Can i just remove or disable oom-killer ? mysql.conf https://pastebin.com/raw/QjqyF52d
Only if you prefer that the whole server just dies with no clear reason, not really worth the effort. Simplest solution is increase memory but that doesn't help much if whatever you're doing is just going to hit up against any limit you raise it to.
Best solution is always to attack it from the ground up. Reduce your MySQL usage by preventing your applications from repeatedly making redundant queries through static caching methods, and keep your database clean of old things that you don't need.
The problem is it just stared today, i never had any problem like this
Seems i am not alone ?? https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842
Openvz?
Probably the host node is out of ram
Most problems do start one day and didn't occur the previous, they all have to have a starting point. You're not alone because this is sysadmin 101, every system administrator who runs basic LAMP/LEMP stacks deals with things like this at some point. You still have to troubleshoot it, you can't run from it. I mean you might be able to, because setting up a new stack elsewhere might iron out a problem for now, but long term you have to troubleshoot.
It started today before few hours, i never see that before in syslog can be RAM defected? how to check? It really no use the all mysql as it cant there is no single script could use that, i watched iotop and htop and nope it just randomly killing mysql, sh, bash etc ... but why? ubuntu bug?
Is this a dedicated server or VM? If VM, what virtualization type?
It's almost never an OS bug, assume that when you've exhausted literally everything else.
It is an online.net server, its just ubuntu server 16.04 with apache2, php, mysql, sure ssh
Use top instead of htop and so shift+m to sort by memory used.
If you're really certain that you're not running out of memory then it may well be that you're the first person I've met that has hit OOM killer for a reason other than oversold OpenVZ or actually running out of memory.
First time for everything right?
I'm pretty sure online.net default install profiles are without swap. Have you tried adding some?
/swapfile is here
Not really, the OOM killer is there to prevent a kernel panic - you want it there. You can exclude processes from it, but that's a terrible idea.
I don't see anything in your MySQL configuration that looks too terribly out of place. I'd recommend installing something like this [1] that records things like resource usage, processes, and MySQL queries to see what exactly is going on. We use it a lot at Rackspace, it's really useful. There'll be a bunch of logs made in /var/log/recap/ that'll contain these details. Simply check the logs around the time the next OOM happens, and you should find your smoking gun.
1: https://github.com/rackerlabs/recap
Is this syssnap racker style?
Yup! This doesn't store the logs in a weird almost-proprietary format, though. Sys-snap at HG was more refined, but I find 'recap' more useful (it's also free to the public!).
I even wrote a tool some of us use internally to provide an easy interface to the logs. It's for an older (non-public) predecessor to 'recap' that's been deprecated though and I haven't had the time to update to support 'recap' yet. If/when that happens I'm going to try to get it merged.
could it be my mysql somehow fcked up, innodb tales got broken and when inserting new things it wont insert and just hanging out then thing and using more and more ram on insert into broken database? but now how to fix innodb myisam i can fix but innodb?
i did this:
screen -S mysql_repair
in screen i did --> mysqlcheck predb -uroot -p
after a while screen got terminated lol, like wtf
Sounds to me like you are running a site which was likely getting attacked; xmlrpc.php POST requests, wp-login.php POST requests, or even just a script on your site that when executed many times results in using all your memory resources up. Usually these types of attacks take place quickly and their intention is to make your server reach a point where OOM kills the services running the site and then they stop.
That said, it doesn't sound like to me that you spent any time actually optimizing the services you are running and likely have unrealistic settings for the amount of memory you have available to Apache/MySQL. I suggest as a first step you get 'mysqltuner' and run it and adjust things. This optimization program will also give you an idea of how much resources would be used if all settings you have in place are maxed out -- it is very possible you have MySQL set to be able to use way more memory than is available on your server.
Past that I would make sure in your Apache config that you have a realistic amount of workers set and don't have it set so high that if the server was attacked that Apache would use a lot of resources quickly.
TL;DR:
it sounds like you need to review , optimize your services on the server and review the site logs to make sure there is not abuse against the sites on the server causing it to quickly amp up and run out of resources. Once optimized correctly the site should be able to handle its max workers/connections without OOMing. You can test this with tools like 'siege' for example.
my 2 cents.
Cheers!
What's the count on that table?
InnoDB is usually either on or off, it's pretty fragile. If it can't read the InnoDB files due to corruption, MySQL will either not start at all, or turn the InnoDB engine off. You can confirm by running the query 'show engines;'.
InnoDB also doesn't get repaired like normal MyISAM data. MySQL will try to do its own InnoDB recovery on startup, it's an automatic thing. If I were to guess, I'd say the screen probably got OOM killed also. mysqlcheck can be pretty resource intensive.
There has to be something eating all of this memory or the kernel doesn't have as much memory as it thinks it does (eg: OpenVZ with stupid-low SHM limits or something really weird set in '/etc/security/*').
Do you recall lowering the value of 'swappiness' (sysctl -a | grep swappi)? Perhaps something isn't getting swapped out because the kernel is trying to keep everything in memory.
It's unlikely though, I'd say MySQL is the likely cause. To me it sounds like the OOM killer solved the problem - when MySQL gets killed, memory usage is 1/32GB. I'd try reverting your MySQL configuration and start from there. After that, install 'recap' and check the logs the next time it gets killed and you'll hopefully see a process in the 'ps' logs with high memory usage.
i need to go to work, will try later tomorrow thx all for replies, will try set lower limits on mysql.conf and apache
also:
vm.swappiness = 60
I'm on break at work right now! Have fun, only a couple hours left in my day :} Swappiness is good, so we can rule that out!
Whatever you do, please don't sacrifice child. It's not worth it, trust me.
Vote is at 1 for killing parent, 0 for sacrificing child.
But uptime!
"Sorry, son, but daddy only bought a 64MB VPS and it's either you or adding a swapfile..."
The trials and tribulations of being brought up in OpenVZ, when those rich capitalist KVM bastards have oodles and oodles of space to swap out their spawn!
Check your limit privilege!
It seems much worse, the server killed itself in the night, no network, no ssh, when i do reboot then i cant ssh too, but rescue mode works, any way to fix boot? as probably boot is hell broken or check if boot isnt broken?
Also in IDRAC -> Virtual Console i see it wont load image: