New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Kernel panic on Hetzner
Hi. I recently reinstalled my Ubuntu server which had an awesome uptime. I installed Debian.
Now it has been 2 times my Debian getting kernel panic. Both happen around the same time, 0420-0425 GMT+8 based on monitoring tool reports. Asimple reboot will take the server up again.
This is all I got from KVM console. There is no error in logs inside /var/log
either.
Can anyone tell me what I need to do? Since the server run normally again after a reboot, and no logs can help, I think this will be hard. I just started to love Debian Buster and now this happen.
Thanks.
Comments
The important part of a kernel panic is the stack trace is above your output.
I recommend setting up netconsole to capture it.
Contact hetzner for verification that this was not caused by hardware issues.
Did you check kern.log and
dmesg
? It may not be in syslog but sometimes you'll find it in kern.logIt's a panic. That kernel be dead.
netconsole is what he needs.
Oh yeah, you're right. I skimmed the post too quickly and thought it may have been a CPU bug / softlock (which is usually written to kern.log).
Two questions if I may ask:
1. What are the server specs?
2. Why did you abandon Ubuntu?
Thanks. This looks complex. I'll take my time reading the tutorial.
Do I need to wait for the kernel panic to occur?
No. You need to provide them clear logs. Previously there was a reports about random kp's on specific line of hetz servers.
By logs, which logs you are talking about? I checked every single logs in
/var/log
and I can only see the log before the system went offline and after the server booted. There is a gap between them where the server went offline.I believe you have to turn on kernel dumping in order for it to save a log of the time of the crash, otherwise it won't save it at all.
See: https://www.linuxjournal.com/content/oops-debugging-kernel-panics-0
I actually> @Trav said:
Thanks. I am trying netconsole right now since it look much simpler on Ubuntu wiki page.
Most likley your problem is caused by r8169 driver try to downgrade to r8168 buy following this tutorial :
https://community.hetzner.com/tutorials/installing-the-r8168-driver?title=Installation_des_r8168-Treibers/en
Thanks! I am not sure what is the issue since the kernel panic hasn't occurred yet since I set the netconsole up. That is weird because before this it happened twice around the same time at different day. I'll refer to this link if that is the case.
Seems that the bug mentioned in that article is for kernel <4.17. Mine is
4.19.0-13-amd64
. Should I try it anyway?I captured the log. This is the trimmed version, with IP removed. Log is 3.6MiB so I think having it in a file is better.
https://mega.nz/file/GBx0la5T#FYVhjy2nUeL8zJksCPTyUmFqMyghXDpFBg13cPvEOoI
Logs earlier than the timestamp in log above only contains
so I removed that.
Does anyone know what is going on in this log? I don't understand at all. Should I forward this to Hetzner?
I forgot to update. I asked Hetzner to do a full hardware check up and this is their replies after 10hours of checking.
That was 4 days ago. In these 4 days, I had no kernel panic. I am not sure if that is the cause or what. but I will continue watching.
You can use the command 'sensors' to monitor it.
unfortunately, even large hoster provider have difficulty some times.
But, Hetzner are good provider, most uptime.
Also, try to force storage check with fsck,
login to hoster console and write grub fsck values:
fsck.mode=force fsck.repair=yes
then boot.