Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Out of memory: Kill process 27935 (mysqld) score 6 or sacrifice child
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Out of memory: Kill process 27935 (mysqld) score 6 or sacrifice child

xrzxrz Member
edited April 2017 in Help

WHen i do htop i use 1GB from 31 GB so how ?

from syslog:

https://pastebin.com/raw/b0uz0qjX

Comments

  • jlayjlay Member
    edited April 2017

    Something's invoking the OOM killer due to memory usage, MySQL gets targeted because it has the highest score (presumably because it's the single largest process) at the time. The value you are seeing might be after the kill occurred, or there may be something trying to allocate large amounts of memory, exhausting it, and failing (eg: MySQL buffers/caches or spurts of large number of httpd/php-fpm processes).

    What does your MySQL configuration look like?

    Thanked by 2xrz Clouvider
  • xrzxrz Member

    Can i just remove or disable oom-killer ? mysql.conf https://pastebin.com/raw/QjqyF52d

  • jarjar Patron Provider, Top Host, Veteran

    @xrz said:
    Can i just remove or disable oom-killer ? mysql.conf https://pastebin.com/raw/QjqyF52d

    Only if you prefer that the whole server just dies with no clear reason, not really worth the effort. Simplest solution is increase memory but that doesn't help much if whatever you're doing is just going to hit up against any limit you raise it to.

    Best solution is always to attack it from the ground up. Reduce your MySQL usage by preventing your applications from repeatedly making redundant queries through static caching methods, and keep your database clean of old things that you don't need.

  • xrzxrz Member
    edited April 2017

    The problem is it just stared today, i never had any problem like this :/

    Seems i am not alone ?? https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842

  • Openvz?
    Probably the host node is out of ram

  • jarjar Patron Provider, Top Host, Veteran
    edited April 2017

    @xrz said:
    The problem is it just stared today, i never had any problem like this :/

    Seems i am not alone ?? https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842

    Most problems do start one day and didn't occur the previous, they all have to have a starting point. You're not alone because this is sysadmin 101, every system administrator who runs basic LAMP/LEMP stacks deals with things like this at some point. You still have to troubleshoot it, you can't run from it. I mean you might be able to, because setting up a new stack elsewhere might iron out a problem for now, but long term you have to troubleshoot.

    Thanked by 1bashlyk
  • xrzxrz Member

    It started today before few hours, i never see that before in syslog can be RAM defected? how to check? It really no use the all mysql as it cant there is no single script could use that, i watched iotop and htop and nope it just randomly killing mysql, sh, bash etc ... but why? ubuntu bug?

  • jarjar Patron Provider, Top Host, Veteran

    Is this a dedicated server or VM? If VM, what virtualization type?

    It's almost never an OS bug, assume that when you've exhausted literally everything else.

    Thanked by 1xrz
  • xrzxrz Member
    edited April 2017

    It is an online.net server, its just ubuntu server 16.04 with apache2, php, mysql, sure ssh

  • jarjar Patron Provider, Top Host, Veteran

    Use top instead of htop and so shift+m to sort by memory used.

    If you're really certain that you're not running out of memory then it may well be that you're the first person I've met that has hit OOM killer for a reason other than oversold OpenVZ or actually running out of memory.

    First time for everything right? :)

    Thanked by 1xrz
  • I'm pretty sure online.net default install profiles are without swap. Have you tried adding some?

    Thanked by 1xrz
  • xrzxrz Member
    edited April 2017

    /swapfile is here

    sudo swapon -s
    Filename Type Size Used Priority
    /dev/sda3 partition 524284 485484 -1
    /dev/sdb3 partition 524284 9208 -2
    /swapfile file 4194300 0 -3

  • jlayjlay Member
    edited April 2017

    @xrz said:
    Can i just remove or disable oom-killer ? mysql.conf https://pastebin.com/raw/QjqyF52d

    Not really, the OOM killer is there to prevent a kernel panic - you want it there. You can exclude processes from it, but that's a terrible idea.

    I don't see anything in your MySQL configuration that looks too terribly out of place. I'd recommend installing something like this [1] that records things like resource usage, processes, and MySQL queries to see what exactly is going on. We use it a lot at Rackspace, it's really useful. There'll be a bunch of logs made in /var/log/recap/ that'll contain these details. Simply check the logs around the time the next OOM happens, and you should find your smoking gun.

    1: https://github.com/rackerlabs/recap

    Thanked by 1xrz
  • jarjar Patron Provider, Top Host, Veteran

    Is this syssnap racker style? :D

  • jlayjlay Member
    edited April 2017

    @jarland said:

    Is this syssnap racker style? :D

    Yup! This doesn't store the logs in a weird almost-proprietary format, though. Sys-snap at HG was more refined, but I find 'recap' more useful (it's also free to the public!).

    I even wrote a tool some of us use internally to provide an easy interface to the logs. It's for an older (non-public) predecessor to 'recap' that's been deprecated though and I haven't had the time to update to support 'recap' yet. If/when that happens I'm going to try to get it merged.

    Thanked by 2xrz jar
  • xrzxrz Member
    edited April 2017

    could it be my mysql somehow fcked up, innodb tales got broken and when inserting new things it wont insert and just hanging out then thing and using more and more ram on insert into broken database? but now how to fix innodb myisam i can fix but innodb?

    i did this:

    screen -S mysql_repair
    in screen i did --> mysqlcheck predb -uroot -p

    after a while screen got terminated lol, like wtf

  • TheLinuxBugTheLinuxBug Member
    edited April 2017

    Sounds to me like you are running a site which was likely getting attacked; xmlrpc.php POST requests, wp-login.php POST requests, or even just a script on your site that when executed many times results in using all your memory resources up. Usually these types of attacks take place quickly and their intention is to make your server reach a point where OOM kills the services running the site and then they stop.

    That said, it doesn't sound like to me that you spent any time actually optimizing the services you are running and likely have unrealistic settings for the amount of memory you have available to Apache/MySQL. I suggest as a first step you get 'mysqltuner' and run it and adjust things. This optimization program will also give you an idea of how much resources would be used if all settings you have in place are maxed out -- it is very possible you have MySQL set to be able to use way more memory than is available on your server.

    Past that I would make sure in your Apache config that you have a realistic amount of workers set and don't have it set so high that if the server was attacked that Apache would use a lot of resources quickly.

    TL;DR:

    it sounds like you need to review , optimize your services on the server and review the site logs to make sure there is not abuse against the sites on the server causing it to quickly amp up and run out of resources. Once optimized correctly the site should be able to handle its max workers/connections without OOMing. You can test this with tools like 'siege' for example.

    my 2 cents.

    Cheers!

    Thanked by 1gowrann
  • jarjar Patron Provider, Top Host, Veteran

    xrz said: could it be my mysql somehow fcked up, innodb tales got broken and when inserting new things it wont insert and just hanging out then thing and using more and more ram on insert into broken database? but now how to fix innodb myisam i can fix but innodb?

    What's the count on that table?

  • jlayjlay Member
    edited April 2017

    @xrz said:
    could it be my mysql somehow fcked up, innodb tales got broken and when inserting new things it wont insert and just hanging out then thing and using more and more ram on insert into broken database? but now how to fix innodb myisam i can fix but innodb?

    i did this:

    screen -S mysql_repair
    in screen i did --> mysqlcheck predb -uroot -p

    after a while screen got terminated lol, like wtf

    InnoDB is usually either on or off, it's pretty fragile. If it can't read the InnoDB files due to corruption, MySQL will either not start at all, or turn the InnoDB engine off. You can confirm by running the query 'show engines;'.

    InnoDB also doesn't get repaired like normal MyISAM data. MySQL will try to do its own InnoDB recovery on startup, it's an automatic thing. If I were to guess, I'd say the screen probably got OOM killed also. mysqlcheck can be pretty resource intensive.

    There has to be something eating all of this memory or the kernel doesn't have as much memory as it thinks it does (eg: OpenVZ with stupid-low SHM limits or something really weird set in '/etc/security/*').

    Do you recall lowering the value of 'swappiness' (sysctl -a | grep swappi)? Perhaps something isn't getting swapped out because the kernel is trying to keep everything in memory.

    It's unlikely though, I'd say MySQL is the likely cause. To me it sounds like the OOM killer solved the problem - when MySQL gets killed, memory usage is 1/32GB. I'd try reverting your MySQL configuration and start from there. After that, install 'recap' and check the logs the next time it gets killed and you'll hopefully see a process in the 'ps' logs with high memory usage.

    Thanked by 1xrz
  • xrzxrz Member
    edited April 2017

    i need to go to work, will try later tomorrow :) thx all for replies, will try set lower limits on mysql.conf and apache :)

    also:
    vm.swappiness = 60

  • jlayjlay Member

    @xrz said:
    i need to go to work, will try later tomorrow :) thx all for replies, will try set lower limits on mysql.conf and apache :)

    also:
    vm.swappiness = 60

    I'm on break at work right now! Have fun, only a couple hours left in my day :} Swappiness is good, so we can rule that out!

  • Whatever you do, please don't sacrifice child. It's not worth it, trust me.

    Thanked by 1jar
  • jlayjlay Member
    edited April 2017

    @drdrake said:
    Whatever you do, please don't sacrifice child. It's not worth it, trust me.

    Vote is at 1 for killing parent, 0 for sacrificing child.

  • raindog308raindog308 Administrator, Veteran

    drdrake said: Whatever you do, please don't sacrifice child. It's not worth it, trust me.

    But uptime!

    "Sorry, son, but daddy only bought a 64MB VPS and it's either you or adding a swapfile..."

    Thanked by 1Clouvider
  • WSSWSS Member
    edited April 2017

    The trials and tribulations of being brought up in OpenVZ, when those rich capitalist KVM bastards have oodles and oodles of space to swap out their spawn!

    Thanked by 1raindog308
  • WSSWSS Member

    Check your limit privilege!

    Thanked by 1yomero
  • xrzxrz Member
    edited April 2017

    It seems much worse, the server killed itself in the night, no network, no ssh, when i do reboot then i cant ssh too, but rescue mode works, any way to fix boot? as probably boot is hell broken or check if boot isnt broken?

  • xrzxrz Member
    edited April 2017

    Also in IDRAC -> Virtual Console i see it wont load image:

    TFTP Prefix:
    Trying to load pxelinux.cfg/so-me-id
    boot:

Sign In or Register to comment.