Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


kvm node running very odd with raid5
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

kvm node running very odd with raid5

I just installed a second kvm node with 3x2TB in a raid5 config. I'm using solusvm and I have one vps installed on the node and my loads are: 129.26 122.12 91.94. I know the vps isn't doing anything, it's the node. When I remove all vps from the node the cpu loads are normal around 1.50.

When I run top -c I get:

24337 root      20   0 1450m 571m 4584 S 17.0  0.4   5:15.75 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-0/kernel.24325 -initrd /var/tmp/.guestfs-0/initrd.24325 -device virtio-scsi-pc
22022 root      20   0 1449m 526m 4584 S 16.6  0.4   9:17.96 /usr/libexec/qemu-kvm -global virtio-blk-pci.scsi=off -nodefconfig -nodefaults -nographic -machine accel=kvm:tcg -cpu host,+kvmclock -m 500 -no-reboot -kernel /var/tmp/.guestfs-0/kernel.22010 -initrd /var/tmp/.guestfs-0/initrd.22010 -device virtio-scsi-pc

These are the two processes that are killing the CPU.

When I create a vps it starts with an iso mounted but when i try to install a template it doesn't start so I manually start it using:

virsh create /home/kvm/kvmID/kvmID.xml

The vps starts but only shows the iso mounted and not the installed template. I synced my templates with the new kvm node and transferred all my templates and isos to it. I can't get the new node to work, the old one works totally fine though. I installed them the same way, so my conclusion is that it's the raid5 config. Any thoughts?

Comments

  • Is this node at OVH?

  • drserverdrserver Member, Host Rep

    swraid or hw raid ?

  • swraid and yes OVH, that is scary you asked that. Is this common with OVH?

  • rm_rm_ IPv6 Advocate, Veteran
    edited September 2014

    said: so my conclusion is that it's the raid5 config.

    I have to ask you one thing, do you know what are the steps to check and diagnose the current state of operation of Linux software RAID?

    If yes, why did you not try them and attach their results into the first post?

    If no, I hope you can reassure us you aren't actually trying to use this in production, and have no plans whatsoever of selling anyone VPSes situated on those nodes. Just to reinforce the faith in humanity in some of us. Thanks.

  • Perhaps you are using their custom kernel, on SYS we can choose the default kernel and I am sure on OVH will be the same.

  • I should've added the diagnoses for the raid config checks out 100%, so posting anything would be useless. Even soluslabs is stumped on this one as everything looks fine inside solusvm as well.

    One very interesting thing I found with OVH servers and this probably won't come as a surprise but I faulted the raid config on my first attempt and when I repartitioned the drives wiping them clean with a new os install I found that remnants of the last install stayed on the newly partitioned drives.

    The volume group I created on the drives during the first attempt was there AFTER I wiped and repartitioned. I told OVH about this and they said it wasn't possible.

    I apologize rm_ your faith in humanity will have to be reinforced in other ways :)

  • @edan said:
    Perhaps you are using their custom kernel, on SYS we can choose the default kernel and I am sure on OVH will be the same.

    Hmmm... interesting I will check this out. I am using OVH's centos 6.5 install so I suspect yes it is their custom kernal. I found this guide maybe I'll give it a shot, got nothin' to loose my server is unusable as it is right now.

  • @conlustro said:
    Hmmm... interesting I will check this out. I am using OVH's centos 6.5 install so I suspect yes it is their custom kernal. I found this guide maybe I'll give it a shot, got nothin' to loose my server is unusable as it is right now.

    On SYS there is a checkbox to choose the default kernel after we partitioning the drives, this options only available if we choose custom installations.

  • I don't see anywhere to do that in the OVH manager even when I choose a custom install, so I'm guessing it's not available.

  • Shoaib_AShoaib_A Member
    edited September 2014

    @conlustro said:
    Hmmm... interesting I will check this out. I am using OVH's centos 6.5 install so I suspect yes it is their custom kernal. I found this guide maybe I'll give it a shot, got nothin' to loose my server is unusable as it is right now.

    OVH kernels should never be used if you want to host virtual servers using any platform (KVM, XEN, OpenVZ etc). And I had guessed as soon as I read your original post that the issue is most probably because of OVH kernel but I asked if it is an OVH kernel just to confirm :) Based on my 3+ years of experience with OVH servers I would recommend that you reinstall with your OS's native kernel instead of just trying to change the kernel as sometimes doing this could cause kernel panic & OVH would run an intervention rebooting your server with an OVH kernel again :(

    And SYS installer does have this bug that when you reinstall/repartition it only formats the
    / partition.

    Also, just to clear the doubts please post the output of

    # cat /proc/mdstat

    Thanked by 1edan
  • K2Bytes said: And SYS installer does have this bug that when you reinstall/repartition it only formats the / partition.

    So we must destroy another partition first before reinstall it? or?

  • Shoaib_AShoaib_A Member
    edited September 2014

    @edan said:
    So we must destroy another partition first before reinstall it? or?

    If you have a server from SYS you should do always do that until the fix their installer (remember their is no problem with the installer they use at OVH but the one at SYS only has that stupid bug)

    Thanked by 1edan
  • If I do end up reinstalling I will also be running a shred to make sure everything is clear. How can I reinstall a native kernal with ovh when they only have their recipes of linux to choose from? Did I miss something?

    Here is the readout, possible I missed something but everything looks okay to me.

    # cat /proc/mdstat
    Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multi                                    path] [faulty]
    md127 : active raid5 sda3[0] sdc3[3] sdb3[1]
          3681485824 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
          [===================>.]  resync = 97.8% (1801611392/1840742912) finish=6.7                                    min speed=96123K/sec
    
    md1 : active raid1 sdb1[1] sda1[0] sdc1[2]
          102398912 blocks [3/3] [UUU]
    
    unused devices: 
    
  • Everything is fine with RAID so the issue is with Kernel. And you just have to check a box that you will see during the reinstallation of OS from OVH panel

  • conlustroconlustro Member
    edited September 2014

    What I was planning on doing is a netboot then shred, and then install centos again and before I do any work on the new installation change the kernel so that way if I mess up something I can just wipe it again.

  • conlustroconlustro Member
    edited September 2014

    Here are the steps in the OVH panel:



    After the last step it installs, and that's it.

  • What happens when you choose "Install one of your templates" in the first step?

  • It is only a saved partition setup and nothing more. No check box.

  • @conlustro said:
    It is only a saved partition setup and nothing more. No check box.

    As you have the server from OVH, you can always use iKVM to install your desired OS. Download the CentOS NetInstall & then install via iKVM using OVH mirror.

    http://mirror.ovh.net/ftp.centos.org/6.5/isos/x86_64/

  • No need, there it is!:

    It was a step I forgot about. Thanks a lot for your help K2, and everyone else who contributed. I appreciate all of your help. I will post back after I try this again, if it works OVH should stop putting their weird stuff into those kernels.

  • Unfortunately the original kernel did nothing to solve the issue, fun times at OVH for me!

  • drserverdrserver Member, Host Rep

    ok, raid is rebuilding "growing". Optimize your softraid settings:

    http://www.cyberciti.biz/tips/linux-raid-increase-resync-rebuild-speed.html

    After it is done, load will be 0.

  • conlustroconlustro Member
    edited September 2014

    drserver said: ok, raid is rebuilding "growing". Optimize your softraid settings:

    My raid1 servers work fine, but my server with raid5 is the problem one. What settings do you recommend for a raid5? Also, my vps are not even starting on the raid5 server. Thanks for the info.

  • Did you creating the LVM manually?

  • edan said: Did you creating the LVM manually?

    Yes I did, but I did nothing different from my kvm server that is running perfectly right now.

  • drserverdrserver Member, Host Rep

    pm me with your skype i will help you with raid optimization

    Thanked by 1conlustro
  • conlustroconlustro Member
    edited September 2014

    I had to go through paypal to get a refund for the server from ovh but I got it. I got a new one with only two drives and it's working, the only thing is the cpu loads still go a little high when performing an install.

    What is a good raid.speed_limit_min?

    I've just increased my raid.speed_limit_max to 500000KB and it seemed to help the loads a little, should I go higher?

Sign In or Register to comment.