Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Spectre and Meltdown - The what is my provider going to do about it? thread!
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Spectre and Meltdown - The what is my provider going to do about it? thread!

JoeMeritJoeMerit Veteran
edited January 2018 in Providers

So, I'm really curious, what are providers going to be doing regarding patching their host nodes?

Mass reboots? can any of these patches be done without rebooting? (Kernelcare, KSplice etc...)

«134

Comments

  • update

    reboot

    what exactly are you curious about?

  • TheRedFoxTheRedFox Member
    edited January 2018

    live migrate stuff to backup nodes while you upgrade the main nodes?

    Only summerhosts cannot afford this...

    A bit of reduced performance isn't an issue anyways with almost everyone idling :)

  • ramnetramnet Member, Host Rep

    We patched our KVM service yesterday and are patching legacy OpenVZ today.

    Did the update live in place with a reboot after. As our nodes aren't overloaded to begin with, not seeing any performance problems at all.

    Backups nodes is overkill when it takes literally 10 minutes to apply the patch and reboot.

    Sad to see my 2 years of uptime wiped away, but oh well.

    Thanked by 1gestiondbi
  • I would hope that providers will stagger reboots so that I do not have everything die at once ;)

  • ramnetramnet Member, Host Rep

    @JoeMerit said:
    I would hope that providers will stagger reboots so that I do not have everything die at once ;)

    Yes, that would be logical.

    Did mine 1 machine at a time for that reason.

  • deankdeank Member, Troll

    In before "I am a host and my server is not rebooting back up after some update!" threads.

    Thanked by 2ramnet vimalware
  • ramnetramnet Member, Host Rep

    @deank said:
    In before "I am a host and my server is not rebooting back up after some update!" threads.

    Haha, you're 100% right. This will show everyone who the morons are claiming to be decent hosts.

    Should be fun to watch.

    Thanked by 1vimalware
  • adlyadly Veteran

    @ramnet glad to hear you’ve patched. Still waiting to see the response from other providers.

  • IonSwitch_StanIonSwitch_Stan Member, Host Rep
    edited January 2018

    As I understand it, there are three issues at play here..

    CVE-2017-5753: Known as Variant 1, a bounds check bypass [Spectre] CVE-2017-5715: Known as Variant 2, branch target injection [Spectre] CVE-2017-5754: Known as Variant 3, rogue data cache load [Meltdown]

    Variant 3 effects Linux hosts, but as KVM is fully virtualizing the processor, it does not allow the variant to be exploited between guests. Variant 3 is a concern for LXC/Docker/OpenVZ hosts because it does allow side-channel attacks between instances. There is a patch available for this that hosts are applying. It is possible for processes on the hypervisor itself (ie, dom0 space in Xen parlance) to exploit this vulnerability. This is patched via KPTI/KAISER.

    Variant 1 and 2 are currently not mitigated for KVM hosts as I understand, and requires a hardware/microcode update and support for these new IBRT op-codes in KVM to protect the guests.

    Am I incorrect?

  • ramnetramnet Member, Host Rep
    edited January 2018

    @IonSwitch_Stan said:
    Am I incorrect?

    Red Hat is claiming that the kernel patches they published already have mitigations for Variant 1 and Variant 3.

    Furthermore, Red Hat is claiming that the qemu-kvm patches they published already have mitigations for Variant 2.

    So, it is my understanding that for KVM, you are 100% patched now, unless Red Hat is lying.

  • adlyadly Veteran

    @IonSwitch_Stan Variant 3 (CVE-2017-5754) is Meltdown, not variant 1. This is patched by KPTI provided updates are installed. KVM is unaffected at this time.

  • IonSwitch_StanIonSwitch_Stan Member, Host Rep
    edited January 2018

    I had the variants out of order and fixed them according to (https://www.kb.cert.org/vuls/id/584653).

  • are the guest kvm also need to be patched ? or the host is enough ?

  • SpartanHostSpartanHost Member, Host Rep

    @kassle said:
    are the guest kvm also need to be patched ? or the host is enough ?

    Guest VMs need patched too.

    Thanked by 1kassle
  • WSSWSS Member

    Well, @davidgestiondbi is already on it..

    Thanked by 2gestiondbi ariq01
  • @kassle said:
    are the guest kvm also need to be patched ? or the host is enough ?

    Yes you probably should patch your KVM too. I say probably since as long as you are the only user and don't care about privilegue escalation you could take the risk of leaving it unpatched. As soon as any untrusted code/users enter the picture you don't have much choice though imo.

    Thanked by 1kassle
  • mfsmfs Banned, Member
    edited January 2018

    uptimecalypse

    So far, among my providers, only Hetzner, Netcup and Prometeus have issued alerts and/or reboots (Prometeus announced they'll they may retire the XenPower product completely)

    Three other providers I've VPS with (one of these is a OVZ and is vulnerable to a PoC it seems) are probably still waiting a little bit for the dust to settle

    Thanked by 1tr1cky
  • WSSWSS Member

    @mfs said:
    uptimecalypse

    Nggg. Yeah.

    wss@ns3:~% uptime
     11:49:32 up 374 days, 21:34,  1 user,  load average: 0.65, 0.64, 0.63
    
  • perennateperennate Member, Host Rep
    edited January 2018

    IonSwitch_Stan said: Variant 1 and 2 are currently not mitigated for KVM hosts as I understand, and requires a hardware/microcode update and support for these new IBRT op-codes in KVM to protect the guests.

    If you are referencing https://www.qemu.org/2018/01/04/spectre/, I think what they are saying is that although kernel/microcode updates that are already available are sufficient to prevent guests from reading hypervisor memory, qemu/kvm updates are needed to prevent applications in the guest from being able to read the guest kernel memory, because qemu needs to expose some CPU functionality ("the new CPUID bits and MSRs") to the guest kernel for the guest kernel to protect itself.

    I'm not sure if the qemu/kvm updates are available now, but when they do become available, the post says that although a reboot of the host will not be required, the guest does need to be rebooted even if the guest kernel is already updated (live migration is insufficient), because it needs to get the exposed CPU functionality on boot.

    Edit: actually it sounds like an unofficial patch for kvm developed by Intel was already available before the qemu blog post (https://lists.nongnu.org/archive/html/qemu-devel/2018-01/txtfadLGhMEF6.txt); so distributions probably already have it although the blog post indicates that it is not merged in upstream; also the author of the blog post seems to suggest there are some problems with this unofficial patch? (https://lists.nongnu.org/archive/html/qemu-devel/2018-01/msg00867.html)

  • IonSwitch_StanIonSwitch_Stan Member, Host Rep
    edited January 2018

    @perennate

    What I read from that QEMU document is:

    Without patches, Guests cannot read from the Host memory.

    Without patches, Guests cannot read from other Guests memory.

    WIthout patches, Guests can make sidechannel reads from their own memory

    Without patches, Hosts can make sidechannel reads from their own memory

    The Kernel updates fix the last two items. My understanding currently is that Specrte requires microcode updates/new opcodes to be developed, and that those do not currently exist, and there is no a Linux patch to protect against Specrte.

    Again -- lots of information out there -- please correct/challange me if my summary is wrong.

  • NeoonNeoon Community Contributor, Veteran

    XVMLabs did reboots a few minutes back.

  • joepie91joepie91 Member, Patron Provider
    edited January 2018

    IonSwitch_Stan said: I had the variants out of order

    Heh. Hehehe.

    Thanked by 2vimalware layfon
  • perennateperennate Member, Host Rep
    edited January 2018

    IonSwitch_Stan said: The Kernel updates fix the last two items. My understanding currently is that Specrte requires microcode updates/new opcodes to be developed, and that those do not currently exist, and there is no a Linux patch to protect against Specrte.

    Again -- lots of information out there -- please correct/challange me if my summary is wrong.

    I think both Spectre and Meltdown are side-channel reads.

    According to the qemu blog post, guests cannot exploit Meltdown to read hypervisor memory, but CAN use Spectre to read hypervisor memory ("CVE-2017-5715 is notable because it allows guests to read potentially sensitive data from hypervisor memory"). But AFAIK there is a Linux kernel update combined with microcode updates that prevents guests from reading hypervisor memory. The blog post indicates both kernel update and microcode update are necessary even for bare-metal: "Just like on bare-metal, the guest kernel will use the new functionality provided by the microcode or millicode updates".

    However there is no official qemu or kvm patch that prevents guest applications from reading guest kernel memory; but it sounds like distributions are already releasing the unofficial patch which the Paolo person (who wrote the blog post) said was written by Intel. So e.g. in Red Hat's updates everything is probably already resolved, as ramnet said.

    Edit: ok I think you're right about the microcode, but then I'm confused by the blog post wording o.o

    Edit2: so http://metadata.ftp-master.debian.org/changelogs/non-free/i/intel-microcode/intel-microcode_3.20171215.1_changelog indicates Debian has microcode updates that resolve CVE-2017-5715, RHEL/others seem to have the same update

  • @SpartanHost said:

    >

    Guest VMs need patched too.

    @mksh said:

    Yes you probably should patch your KVM too. I say probably since as long as you are the only user and don't care about privilegue escalation you could take the risk of leaving it unpatched. As soon as any untrusted code/users enter the picture you don't have much choice though imo.

    hope the performance penalty effect not doubled then

  • @ramnet said:

    @IonSwitch_Stan said:
    Am I incorrect?

    Red Hat is claiming that the kernel patches they published already have mitigations for Variant 1 and Variant 3.

    Furthermore, Red Hat is claiming that the qemu-kvm patches they published already have mitigations for Variant 2.

    So, it is my understanding that for KVM, you are 100% patched now, unless Red Hat is lying.

    The kernels might contain the patches, however - the microcode has to be there, and it's currently not.

    In another thread, one linked to a debian package which might contain the microcodes for both spectre variants, however these microcodes are yet to be seen on other operating systems and on intels own website.

    So no. People are not patched fully until they've updated the microcode for the CPUs.

  • perennateperennate Member, Host Rep
    edited January 2018

    Zerpy said: In another thread, one linked to a debian package which might contain the microcodes for both spectre variants, however these microcodes are yet to be seen on other operating systems and on intels own website.

    Someone said the Debian ones are the same already released by RHEL.

    Edit: can't find the link now, but https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=886367#21 implies that the microcode updates were published by Intel yesterday (first message says expecting an update, next one saying the updates are pushed to Debian)

    Edit2: I found the link, it also has more info in general -- https://github.com/hannob/meltdownspectre-patches#cpu-microcode

  • @perennate said:

    Zerpy said: In another thread, one linked to a debian package which might contain the microcodes for both spectre variants, however these microcodes are yet to be seen on other operating systems and on intels own website.

    Someone said the Debian ones are the same already released by RHEL.

    In that case, they'll not fix spectre for a bunch of CPUs such as:
    i7-6700k
    E3-1230
    E3-1230v6
    E5-1630v2
    E5-1630v3
    E5-1630v4
    E5-1650v3
    E5-1650v4
    E5-2620v3
    D-1540

    Because none of those CPUs even after updated microcodes and kernels has ibpb and ibrs enabled:

    $ cat /sys/kernel/debug/x86/{ibrs_enabled,ibpb_enabled}
    0
    0
    
    Thanked by 1perennate
  • perennateperennate Member, Host Rep
    edited January 2018

    Zerpy said: In that case, they'll not fix spectre for a bunch of CPUs such as: i7-6700k E3-1230 E3-1230v6 E5-1630v2 E5-1630v3 E5-1630v4 E5-1650v3 E5-1650v4 E5-2620v3 D-1540

    Because none of those CPUs even after updated microcodes and kernels has ibpb and ibrs enabled:

    Ah, this is with RHEL? The RHEL patch seemed more comprehensive than the Debian one actually since Debian hasn't updated kernel package yet AFAIK. So that is strange... hopefully get more information from Intel/others soon.

  • ramnetramnet Member, Host Rep

    @Zerpy said:
    The kernels might contain the patches, however - the microcode has to be there, and it's currently not.

    In another thread, one linked to a debian package which might contain the microcodes for both spectre variants, however these microcodes are yet to be seen on other operating systems and on intels own website.

    So no. People are not patched fully until they've updated the microcode for the CPUs.

    Red Hat released microcode updates also.

    Linux has long had the ability to patch the microcode during OS bootup, unlike certain other OSes which require BIOS updates to do that.

  • perennateperennate Member, Host Rep
    edited January 2018

    ramnet said: Red Hat released microcode updates also.

    Just curious, if you run the command that Zerpy wrote, do you see that these are enabled?

    cat /sys/kernel/debug/x86/{ibrs_enabled,ibpb_enabled}

    Edit: (on RHEL 6 need to do this first: mount -t debugfs nodev /sys/kernel/debug)

    Edit2: and more information here https://access.redhat.com/articles/3311301

    As noted, installing the microcode update for your hardware, if provided by the hardware vendor, is necessary to protect against variant 2. Please contact your hardware vendor for microcode updates.

    Edit3: more commentary here too https://serverfault.com/questions/890904/i-updated-my-centos-7-system-why-is-meltdown-spectre-only-partially-mitigated; at least one person said they are seeing IBRS/IBPB enabled, so I guess the microcode does implement it but only for a very restricted set of CPU models?

Sign In or Register to comment.