MySQL server won't start

lukelarris

Yo,
A brief introduction: I run a 512 MB "droplet" VPS at DigitalOcean, OS is Debian 7 x64, and the MySQL server I use is MariaDB. So far I haven't had any issues with it, but it just mysteriously broke.

Today after about 3 weeks of not opening up the CLI I tried to add a new MySQL database. When the command failed to run repeatedly, I then tried to restart the MySQL service. When it was starting it, it came up as failed.

I've tried reinstalling MariaDB, I've checked the logs, they had nothing in them, and I've rebooted the droplet a few times. No go.

Says it can't connect to the socket, and it's not a permissions issue as far as I can tell.
It's really weird because before today I hadn't touched anything for about 3 weeks, and my sites that use MySQL were working fine.
Any ideas why this is happening? I appreciate any help.

upfreak

Check free space in /tmp.

lukelarris

Got plenty of free space in tmp.

netomx

I got that, check the logs in /var/log/myswl or try to start the service manually. I can't recall on my head, but I readed the /etc/init.d/mysql file to know the actual command, it helped me debug. It is probably a config line

lukelarris

There's only a bunch of mariadb-bin.00000xx files in /var/log/mysql/ and they seem to contain table operation logs in gibberish.

jar

Exact text of the error might be helpful. You could also try an strace of the init script and see if you get more targeted information.

lukelarris

When I try to start the mysql service I get this:
luke@enigma:/var/log$ sudo /etc/init.d/mysql start
[sudo] password for luke:
[FAIL] Starting MariaDB database server: mysqld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed!

Will try to reproduce the socket error.
How do you trace an init script?

jar

@lukelarris said:
When I try to start the mysql service I get this:
luke@enigma:/var/log$ sudo /etc/init.d/mysql start
[sudo] password for luke:
[FAIL] Starting MariaDB database server: mysqld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed!

Will try to reproduce the socket error.
How do you trace an init script?

strace -t /etc/init.d/mysql

netomx

Open /etc/init.d/mysql and search for the exact command

sman

Might want to check if /etc/my.cnf got changed. Maybe try replace it with a default config and see what happens.

I know you said you checked but it could still be a permissions problem. Permissions are the biggest headache with Linux imho. Often don't generate error logs so hard to troubleshoot. Can sometimes be some totally unexpected and seemingly unrelated folder/file like /etc/hosts.

Also might want to try some standard troubleshooting things. Like disabling SELinux and IPtables if enabled.

horas

make sure you have a loopback interface (lo) - whenever your mysql listening on

lukelarris

@jarland said:
strace -t /etc/init.d/mysql

"Command not found: strace"
What package do I need to install?

@sman said:
Might want to check if /etc/my.cnf got changed. Maybe try replace it with a default config and see what happens.

I know you said you checked but it could still be a permissions problem. Permissions are the biggest headache with Linux imho. Often don't generate error logs so hard to troubleshoot. Can sometimes be some totally unexpected and seemingly unrelated folder/file like /etc/hosts.

Also might want to try some standard troubleshooting things. Like disabling SELinux and IPtables if enabled.

Wouldn't purging MariaDB and installing it again replace the my.cnf though?

Alright, well I'll play around with the permissions some more. I don't know where else to check though.

I don't remember if I installed IPtables, how would I check that?

jar

Package should install as just strace if memory serves.

lukelarris

Managed to trace it to a bad my.cnf file, which worked fine 3 weeks ago. What's wrong here?
http://pastebin.com/vUUhzWvF

Grimley

innodb_flush_method =
No value?

innodb_additional_mem_pool_size=8m <- Does it work like that? Shouldnt it be like size = 8m?

exussum

do a sudo mysqld and see the output ?

lukelarris

@Grimley said:
innodb_flush_method =
No value?

innodb_additional_mem_pool_size=8m <- Does it work like that? Shouldnt it be like size = 8m?

Not sure what you're talking about, innodb_flush_method has a value:
innodb_flush_method = O_DIRECT

And the spaces don't matter.

alegeek

apt-get install strace

Then provide the output of: strace /etc/init.d/mysql start

jar

@lukelarris said:
Managed to trace it to a bad my.cnf file, which worked fine 3 weeks ago. What's wrong here?
http://pastebin.com/vUUhzWvF

It said bad my.cnf at the end? I guess you could output ls and lsattr but I doubt anything changed in that direction.

lukelarris

@jarland said:
It said bad my.cnf at the end? I guess you could output ls and lsattr but I doubt anything changed in that direction.

No, on a whim I used the default my.cnf and bam everything worked.

jar

@lukelarris said:
No, on a whim I used the default my.cnf and bam everything worked.

Oh interesting. My eyes aren't catching the issue in the original. Glad it's working for you though.

netomx

Use your bad my.cnf file and do this:

/usr/bin/mysqld_safe

jcaleb

default my.cnf is slow

lukelarris

@netomx said:
Use your bad my.cnf file and do this:

/usr/bin/mysqld_safe

Wow, thanks. Checked the log generated by that and apparently the "skip-locking" flag in the config was causing the problems.

Thanks guys!

netomx

I told you before to do that, but you didn't do it -.-

lukelarris

@netomx said:
I told you before to do that, but you didn't do it -.-

I don't see it in this thread other than that one post.

netomx

@netomx said:
Open /etc/init.d/mysql and search for the exact command
@lukelarris said:
I don't see it in this thread other than that one post.

lukelarris

Lol, oh well.

netomx

Maybe i wasnt clear enough