Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Data corruption on Kvm/Xen/Openvz vps container
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Data corruption on Kvm/Xen/Openvz vps container

Let's suppose that we have three vps containers with untouched operating system on different virtualizations such as KVM/Xen/OpenVZ
Each vps containers are hosted with different providers.

After a month all of the vps containers went down. All vps providers has reported that data on vps was corrupted for unknown reasons. Providers didn't find the issue on their side as they said that they are using Raid 10 setup or etc.

What's the possible reason that data was corrupted on all three vps containers ? Vps containers weren't used at all. Is virtualization type makes sense when a data is lost ?

Thank you.

«1

Comments

  • @Jack said:
    I don't really understand your question.

    What's possible reason of data loss. Is this virtualization or hardware issue?

    Thank you.

  • Containers aren't well secured and has been compromised by hacker.

  • SpiritSpirit Member
    edited December 2013

    Hmm.. I experienced something like this with relatively known reputable US OpenVZ VPS provider some week ago. I am not bothered much as it was more or less idle vps (and I am not complaining) but this was their response in case you're interested:

    It appears that your VPS is broken an I am unable to login to it. You will need to reinstall the OS to fix this problem since we cannot even get into the VPS from the hardware node.

    Slightly more detailed anwer to my question what happened:

    No idea. I've never seen it happen before, even the most damaged VPS can be accessed via the hardware node if it turns on but this is the first time where I've never been able to access a VPS while it was powered on. I've made a complete backup of your VPS so if you want to re-install the OS I can put the backup on your new VPS.

    (OpenVZ/SolusVM)
    Server was properly secured, did not host anything and I doubt about intrusion. Host also confirmed: "The host node is running normally. I checked other VPSs on the node and they are operating as expected."
    Provider is competent and node isn't overloaded btw.

    Thanked by 1vRozenSch00n
  • This could be hardware compatibility issue (raid + ssd) or buggy kernel

  • It's most likely the template. We had to drop centos 5 32bit due to template issues. Data corrupts on vps reboot.

    Thanked by 1vRozenSch00n
  • Yup, @jack pointed out this url https://bugzilla.openvz.org/show_bug.cgi?id=2647
    (Debian 7 template - I used)

    Thanked by 1vRozenSch00n
  • [My Fault], playing with IPconfig tried some addon and blocked all the access to the VPS (OpenVZ), so no ssh or anything :o

    But it was fixed for me from support Department.

  • MaouniqueMaounique Host Rep, Veteran

    If you are cut out from ssh you can still access the console. Any decent vps provider will give you console access of some sort.

    Thanked by 1MikeIn
  • @Maounique said:
    If you are cut out from ssh you can still access the console. Any decent vps provider will give you console access of some sort.

    There's also ipv6.

  • MaouniqueMaounique Host Rep, Veteran

    Yeah, but more than half providers seem not to have it...

  • @Jack said:
    OpenVZ I would blame the provider really as you can't really do much but KVM/XEN you can do as much as the provider can do really.

    What if I did'nt even touch the vps. Should I blame all those providers for loosing my data even if there were nothing ?

    @vRozenSch00n said:
    Containers aren't well secured and has been compromised by hacker.

    It's untouched, minimal linux OS. There is no chances that they were compromised and yes it was just an example.

    Spirit said: Hmm.. I experienced something like this

    Exactly, I was pointing to the issue like this.

    @concerto49 said:
    It's most likely the template. We had to drop centos 5 32bit due to template issues. Data corrupts on vps reboot.

    @concerto49 @Spirit Is there any issues with other linux distros and versions?

    @MikeIn said:
    [My Fault], playing with IPconfig tried some addon and blocked all the access to the VPS (OpenVZ), so no ssh or anything :o

    But it was fixed for me from support Department.

    @MikeIn @Maounique I'm sorry but all providers has reported that data on vps containers has been lost. It's untouched, minimal linux OS.

    Thank you guys!

  • MaouniqueMaounique Host Rep, Veteran

    Spirit said: It appears that your VPS is broken an I am unable to login to it. You will need to reinstall the OS to fix this problem since we cannot even get into the VPS from the hardware node.

    That does not mean the data is lost. At the very least can tar everything together and put it some place for the customer to take it, they do not need the OS files, but their data files, so, even if the container no longer boots, data should still be salvageable.

  • So what OS was on these containers? Is it possible that some update auto installed and screwed things (incompatible with the OpenVZ kernel, etc.).

  • @rds100 said:
    So what OS was on these containers? Is it possible that some update auto installed and screwed things (incompatible with the OpenVZ kernel, etc.).

    I'm not sure if updates could cause data loss on different virtualizations and even on different operating system for example ?

  • For different operating systems the updates shouldn't be the common factor, true.

    Thanked by 1alexvolk
  • It happened at different providers all at once, so it can't be hardware.

  • @vRozenSch00n said:
    It happened at different providers all at once, so it can't be hardware.

    Well, it was just an example ;)

  • So far the only common factor i see is you. Is it possible that somehow your workstation was compromised and someone used it to login to all these VPSes and delete them?

  • Yes, its getting interesting and tingling :P

  • @rds100 said:
    So far the only common factor i see is you. Is it possible that somehow your workstation was compromised and someone used it to login to all these VPSes and delete them?

    No, I mean all this didn't happen and was supposed to understand the reason if "the shit happens".

  • @alexvolk said:
    Is virtualization type makes sense when a data is lost ?

    I guess this is the only one left, host machine virtualization type.

  • AnthonySmithAnthonySmith Member, Patron Provider

    Well as this never happened and you have ruled out hacking hardware and software.... its not going to happen in 3 location and 3 os types with 3 virt types at the same time.

    Thanked by 1Maounique
  • OpenVZ has the stuck init issue which requires a node reboot since you cant kill an init process.

    The others it's possible the server crashed and your disks failed a fsck check, or you turned off VNC so of course nobody can really access it. Another one is Debian 7 if you encrypt the drive and it thinks it's on a new machine it requires a root password to finish booting.

  • data corruption is a known issue with xen, openvz, kvm
    the only virtualization software that does seem to not randomly corrupt is VMWare.

  • AnthonySmithAnthonySmith Member, Patron Provider

    @Mark_R said:
    data corruption is a known issue with xen, openvz, kvm
    the only virtualization software that does seem to not randomly corrupt is VMWare.

    huh?

    Well After 4+ years of Xen I can say I have never faced or found a known issue that corrupts data, KVM uses essentially the exact same methods and OpenVZ has no storage specific methods, in fact for KVM+Xen+OpenVZ there is nothing specific to the virt type that handles the storage so to say it is a known issue is to say that Linux itself has a known issue that causes data corruption is its either using LVM or the flat file system of the host OS.

    VMware on the other hand has a propitiatory storage system to which I absolutely have seen corruption on due to snapshot failures, driver related issues for raid cards etc and it is much harder to recover from due to its propitiatory nature.

  • @AnthonySmith - It's possible to corrupt inodes/data on Xen/KVM simply by initializing several requests to resize the main disk at once and then rebooting the virtual server several times to mess up the disk resizing process.

  • GreenValueHost said: It's possible to corrupt inodes/data on Xen/KVM simply by initializing several requests to resize the main disk at once and then rebooting the virtual server several times to mess up the disk resizing process.

    If you want to shoot yourself in the foot, isn't it easier to just cat /dev/zero > your disk?
    I don't see how this is Xen or KVM fault.

  • Oh.. I thought we were listing ways to corrupt data .. maybe I should have read the entire thread

    My bad sorry

    Thanked by 1AnthonySmith
  • AnthonySmithAnthonySmith Member, Patron Provider

    @GreenValueHost said:
    Oh.. I thought we were listing ways to corrupt data .. maybe I should have read the entire thread

    My bad sorry

    haha well yes, if anything that re-enforces my point, that is not Xen/KVM/OpenVZ specific.

  • @Mark_R said:
    data corruption is a known issue with xen, openvz, kvm
    the only virtualization software that does seem to not randomly corrupt is VMWare.

    Probably your provider fooled you :p

Sign In or Register to comment.