New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Xen or KVM VPS "rescue" best practices?
Let's suppose I have upgraded a kernel in my VPS, and there is a problem with the new kernel, so that the OS no longer boots. What are my options to recover at this point. With the two Xen providers I used so far, the only option seemed to be "File a ticket and ask them to change back Default=value in /boot/grub/grub.conf for you". Does pyGrub even show any sort of a kernel selection menu to Serial Console? I don't think it does.
So are there any providers with support for booting your VPS into a rescue ISO? I know some KVM providers had that (which ones?), but what about Xen?
Comments
1) you franned it, good job sir
2) You could see if they can put you in HVM mode, or if you have the option?
I know when I used xenserver and I botched kernel updates that I'd have to put things into HVM mode, fix it, then put it back into PV.
Francisco
What? Oh you mean I screwed up a VPS and that's why I'm asking. Nope, not yet I am going along with the Debian -bpo kernels, they are pretty stable and work every time, but 1) I don't feel alright when I have to cross my fingers every time my 'production' VPS goes down for a reboot, fearing whether or not will it come up, and if not, will I have to wait on a ticket; 2) want to try a third-party 3.4.1 kernel right now, and REALLY don't feel at ease doing so, without a solid recovery procedure available.
Pretty sure both of those providers use PV nodes, and asking for switching to HVM temporarily will also involve waiting for the provider's reply (i.e. not something I could immediately do myself).
Can't you use VNC to boot into single user mode to fix whatever you broke?
Not with PV.
If it was HVM then sure and he wouldn't need to open this thread
Francisco
Get a KVM VPS. Sorted.
If it's Xen PV just mount the LVM on your host node and change the boot order for the grub.conf. I had to do this the other day actually.
He doesn't have access to the host node and also said he doesn't want to wait for the provider, KVM seems the only easy solution to me.
If I were to design a recovery solution, I'd add a function to the control panel, something like "boot rescue system", where a client's VPS would be stopped, then a minimal OS image is created for them with their VPS IP address in a separate volume (can be even 1-2GB in size), then it is booted, and the actual VPS image is mounted into /mnt/something/. So that the client can then ssh into the recovery system and amend what's needed in their actual VPS to make it boot, or even chroot into that FS tree it to use apt-get, etc.
This is certainly not impossible even with Xen PV. I kind of hoped someone will say "oh there's this provider who implements [something like the above]". And then we could shame all other providers for not following suit. :P
Oh, there's this provider who implemented this.
If SolusVM allowed it more people would offer it. I know I would.
In XenVZ's control panel you have the option of enabling/disabling pygrub at will, so if something messes up you can quickly reboot with the default kernel.
Which is great n_n
GRUB has a fallback mechanism which is useful when testing a new kernel: http://www.gnu.org/software/grub/manual/legacy/Booting-fallback-systems.html
However on Xen PV you will find pyGrub and not GRUB, I doubt it supports the same mechanism.
You can instead use PvGrub to have the normal Grub inside the Xen domU. That's actually what the VPS-provider I named earlier above does.
I know Linode has the option to boot into a recovery image, sucks SolusVM dosen't offer anything like that.