Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Register

Quick Links


Categories

In this Discussion

› › MyCustomHosting Down ? (Resolved. Issues? Open a ticket first!)
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

MyCustomHosting Down ? (Resolved. Issues? Open a ticket first!)

FredQcFredQc Member
edited May 2015 in Providers

All my servers at MyCustomHosting are down since 4PM EST, their web site MyCustomHosting.net is down also...

What's going on @MCHPhil ?

«13456789

Comments

  • TrafficTraffic Member

    Mine's online.

    Thanked by 1camarg
  • sleddogsleddog Member

    @Traffic said:
    Mine's online.

    In Montreal?

  • TrafficTraffic Member

    @sleddog Yep, OVH Beauharnois

    Thanked by 1TheHackBox
  • FrankZFrankZ Veteran
    edited May 2015

    @FredQc - I received a notice about a server reboot re patching the exploit at 19:07 UTC. My VPS when down shortly after, came back up 5 minutes later, and has been running fine since. I am on node 5.

    Thanked by 1Traffic
  • sleddogsleddog Member

    @Traffic said:
    sleddog Yep, OVH Beauharnois

    Good. Then maybe he's just got a node down with his website and my VM :)

    Thanked by 1Traffic
  • FredQcFredQc Member
    edited May 2015

    Hum... sorry guys but everything is down on my side... Yes, in Montreal.

  • DroidDroid Member

    Same here. Everything down.

  • tcpdumptcpdump Member

    CA-VPS2 and CA-VPS4 are down

  • TrafficTraffic Member

    CA-VPS5 here and online.

  • rokokrokok Member

    damn mine down too

  • joereidjoereid Member

    200+ day uptime reset with a vengence...

    I'm really glad I took Phil's advise to "prepare" for the reboot by taking a few last minute backups. I haven't submitted a ticket because I'm sure he knows there is a problem and I'd rather not add to his frustration.

    http://mycustomhosting.com/network-status.php (ouch)

  • rokokrokok Member

    I hate when provider updating and not prepared informing users first...

    No email, no news...

  • FredQcFredQc Member

    @rokok said:
    I hate when provider updating and not prepared informing users first...

    No email, no news...

    Same thing here...

  • FrankZFrankZ Veteran
    edited May 2015

    On Wed, 13 May 2015 19:07:35 +0000 I received this email from MCH

    Good afternoon,

I need to inform you we will be required to reboot the nodes handling KVM servers shortly.  This is an emergency security update.  Please make arrangements, sorry for the short notice!

My Custom Hosting

    I am sure that all issues will be resolved as soon as possible. I expect that Phil will not answer here until he has done everything possible to resolve the problems first.

    EDIT: IME, MCH will send follow-up emails to affected customers.

  • shrubblesshrubbles Member
    edited May 2015

    I think Phill needed to restart because of Venom exploit.

  • youknownothingjonyouknownothingjon Member

    Got this email from My Custom Hosting:

    There has been an issue with the SSD cache on CAVPS2, CAVPS3 and CAVPS4. At this time the extent of the issue is unknown.

  • rokokrokok Member
    edited May 2015

    I'm not receiving email like FrankZ, but just receiving mail like above me now

  • joereidjoereid Member

    Just got the same email. Sucks that all those nodes have the same issue...

  • rokokrokok Member

    Kinda bad habit for *most provider, act first and expecting lowendhelpdesk thread open

    Thanks for email update.

  • jaypeesmithjaypeesmith Member
    edited May 2015

    Edit. Didn't see that someone already posted the reason for the extended outage.

  • xaitmixaitmi Member

    All down.

  • xaitmixaitmi Member

    Shared hosting is timing out. Keeps going on and off.

  • sleddogsleddog Member

    Looks like SRV4 is up now on the status page. That's progress... Go Phil.

  • smile93smile93 Member

    Mail Received

    Good evening,

There was a catastrophic failure during the maintenance window.  Two distinct things occurred.  They are outlined below.

First, our current restart procedure for a node.

virsh list to grab all running VM's.
virsh shutdown each VM.
shutdown -r now on node.

CA-VPS2 + CA-VPS3
After performing the restart procedure (outlined above) the machines hung.  I gave the machines 3 and a half hours to come back or do anything.  At which point they were rebooted physically.  This caused an issue with the SSD cache destroying the VM's data volume group.
CA-VPS4
After performing the restart procedure (outlined above) the machine restarted clean but experienced a SSD disk failure, the disk was no longer showing in the machine or in BIOS.  OVH has replaced the disk.

Rebuilds on these servers have already begun.  Shared hosting in CA was affected, not by disk failure but it is hosted via CA-VPS3.  No shared hosting or reseller hosting data was lost.

I will not be leaving my post until things are running again.  Credits will be handled at the conclusion.  Extremely sorry for the inconvenience!

Phil Robinson
  • joereidjoereid Member

    So...does that mean all VPS data on 2 and 3 is gone? Sigh...

  • DroidDroid Member

    joereid said: So...does that mean all VPS data on 2 and 3 is gone? Sigh...

    Wanting to know the same.

  • FredQcFredQc Member

    I too received the last two mails. Now, that doesn't look good, heh...

  • blackblack Member

    It's the correct shutdown procedure as far as I'm aware. Everything afterwards is just unfortunate :(

  • SilvengaSilvenga Member

    Bad things happen sometimes. @MCHPhil did handle it well however. Good chance to upgrade my system on a clean vm. :)

  • rokokrokok Member

    looks like all kvm node up now, no miracle to get our data back?

This discussion has been closed.