Down - OVH - SBG - Lots and lots of tears.

xaoc

This: http://travaux.ovh.net/?do=details&id=49471&;

hellb0y

The end is neigh :~

xaoc

@hellb0y said:
The end is neigh :~

It is.

http://travaux.ovh.net/?do=details&id=49484&;

WSCallum

Ouch, major fire at SBG2 - Not good.

yoursunny

@WSCallum said:
Ouch, major fire at SBG2 - Not good.

Blame the FIRE THREAD on LET.

K4Y5

We are currently facing a major incident in our Strasbourg datacentre, with a fire declared in the SBG2 building.
Firefighters intervened immediately on the spot but were unable to control the SBG2 fire.
As a precautionary measure, the electricity was cut off on the whole site, which impacts all our services at SBG1, SBG2, SBG3 and SBG4.
If your production is in Strasbourg, we recommend that you activate your Business Recovery Plan.
All our teams are mobilized alongside the firefighters.
We will keep you posted on this as soon as we have more information.

Oh well..
/me looks at the LE entrepreneurs losing billions due to lack of disaster recovery plans in place.

TL;DR Version:
Always, always and ALWAYS have back-ups in place. Depending on criticality, you may want to setup hourly, daily or weekly back-ups and store the files off-site. Preferably, at a different location / DC and provider.

sin

Ouch.

Moofie

Maybe this will finally get me to setup backups on my servers lol

JabJab

Planned! #insurance scam

I guess I will need to reinstall my VPS soon. I really hope that all of this is just a 'procedure' and not whole building is on fire...

No juicy photos/drone videos from site? Can't be that everyone is sleeping there, get on some French news page!

valk

Yikes! Literally

Jasonhyperhost

Octave OVH CEO has posted on his twitter the below statement /cry

https://twitter.com/olesovhcom/status/1369478732247932929

coolice

Plan is activated but IP movement takes forever -

You cannot move this IP, as it is already in the process of being transferred. -

Does someone manage to move IPs out ?

WSCallum

@JabJab said:

Planned! #insurance scam

I guess I will need to reinstall my VPS soon. I really hope that all of this is just a 'procedure' and not whole building is on fire...

No juicy photos/drone videos from site? Can't be that everyone is sleeping there, get on some French news page!

It’s SBG2 and seems to have affected whole of SBG2 according to octave on twitter, 1,3,4 i assume will be back up when its safe to restore power

lentro

holy shit. that's a huge nightmare situation.

JabJab

@WSCallum said: , 1,3,4 i assume will be back up when its safe to restore power

Problem is that those all buildings are connected. Bah, you can even say it's one building.

https://www.google.com/maps/place/9+Rue+du+Bassin+de+l'Industrie,+67000+Strasbourg,+Francja/@48.5858015,7.7975347,193m/data=!3m1!1e3!4m5!3m4!1s0x4796c8daab82f9af:0x538b85a8eb23fadb!8m2!3d48.5858131!4d7.7973832

JabJab

https://twitter.com/olesovhcom/status/1369504527544705025

Update 5:20pm. Everybody is safe.
Fire destroyed SBG2. A part of SBG1 is destroyed. Firefighters are protecting SBG3. no impact SBG4.

WSCallum

@JabJab said:

@WSCallum said: , 1,3,4 i assume will be back up when its safe to restore power

Problem is that those all buildings are connected. Bah, you can even say it's one building.

https://www.google.com/maps/place/9+Rue+du+Bassin+de+l'Industrie,+67000+Strasbourg,+Francja/@48.5858015,7.7975347,193m/data=!3m1!1e3!4m5!3m4!1s0x4796c8daab82f9af:0x538b85a8eb23fadb!8m2!3d48.5858131!4d7.7973832

Interesting, never looked at the plans so didn't know they were almost wall to wall connected, assumed they'd be separate "buildings" in a small area as such.

That'll make it more interesting then...

serv_ee

@WSCallum said:

@JabJab said:

@WSCallum said: , 1,3,4 i assume will be back up when its safe to restore power

Problem is that those all buildings are connected. Bah, you can even say it's one building.

https://www.google.com/maps/place/9+Rue+du+Bassin+de+l'Industrie,+67000+Strasbourg,+Francja/@48.5858015,7.7975347,193m/data=!3m1!1e3!4m5!3m4!1s0x4796c8daab82f9af:0x538b85a8eb23fadb!8m2!3d48.5858131!4d7.7973832

Interesting, never looked at the plans so didn't know they were almost wall to wall connected, assumed they'd be separate "buildings" in a small area as such.

That'll make it more interesting then...

Dont think "interesting" is the right word right now..

lentro

@WSCallum said: almost wall to wall connected

P sure they are basically a huge building.

https://cdn.baxtel.com/data-center/ovh-strasbourg-campus/photos/OVH-Strasbourg-Campus-OVH-Strasbourg-Campus-showing-all-3-Buildings-2019.png

Vova1234

This is sudden. In SBG, mainly game servers dropped out: i7-4790k, Ryzen 5/7, i7-6700k, i7-7700k.

There was already a global incident in 2017. But for everything to burn out in 2021.

JabJab

http://status.ovh.com/vms/index_sbg2.html

Nice monitoring you got there /s

servarica_hani

if you are affected and urgently need VPS for few days you can order from us and cancel in 7 days and you will get full refund
If you need more time pm me will figure something out

if you are looking for storage vps also pm will provide it although at this moment the performance is not great as we are doing major moves to update our storage servers

WSCallum

Anyone who has servers in SBG2 it's pretty much guaranteed toast, SBG1 partial destroyed basically according to Oles on Twitter:

Update 5:20pm. Everybody is safe.
Fire destroyed SBG2. A part of SBG1 is destroyed. Firefighters are protecting SBG3. no impact SBG4.

valk

Hope it doesnt escalate to SBG3, or I piss my pants!!!!

xaoc

@WSCallum said:
Anyone who has servers in SBG2 it's pretty much guaranteed toast, SBG1 partial destroyed basically according to Oles on Twitter:

Update 5:20pm. Everybody is safe.
Fire destroyed SBG2. A part of SBG1 is destroyed. Firefighters are protecting SBG3. no impact SBG4.

5:20pm, eh?

SWN_Michael

If anyone needs assistance, please reach out to me. I know we're very far from this OVH datacenter location, but if there's anything that can be done, I'm glad to help.

serv_ee

Guess the French gov. needs to open up their wallet again after this.

nurex

The word "destroyed" really shocked me. Hope everyone is OK.
I recall there was a mega power outage several years ago, and today another major incident hits. Sad enough.

smallbibi

ovh needed better firewalls

serv_ee

Reading the Twitter comments just...smfh. People really are idiots.

xaoc

@serv_ee said:
Reading the Twitter comments just...smfh. People really are idiots.

NetDynamics24

If anyone is affected and needs temporary backup storage or anything else, please contact us. We are willing to offer some of our services free of charge for some period.

rcxb

coolice

Still moving failover IPs is stuck

miu

bad dream this, i already restore affected services from backups to elsewhere.. I cant understand how can whole datacenter be destroyed by fire. Epic failure..

fazar

oh wow, this should be a nightmare for tenants at SBG. 😔

BigB

Rip my vps

sdglhm

Hope staff made it out safely. It seems like SBG2 is fully gone and if you didn't have backups, you're toast.

http://www.taobackup.com/

xaoc

https://twitter.com/olesovhcom/status/1369535787570724864

Falzo

wonder what's there to actually burn in these tin cans... with what did they isolate?

if you look at this vid don't get your hopes to have anything from "SBG2" recovered at all:

https://twitter.com/_EstebanL/status/1369540663876980737

Konbu

I hope everybody is safe.
Looking at the pictures, their data center was more of a truck container than a building.Σ(ﾟдﾟ；)

corbpie

That’s why my favourite CSGO server was down sadpikachu.jpg

Chex2

@Konbu said:
I hope everybody is safe.
Looking at the pictures, their data center was more of a truck container than a building.Σ(ﾟдﾟ；)

Yeah it's basically shipping containers. I recall they were marketing how cheap it was to build because of this. Very worrying the fire suppression either failed or was not sufficient.

sdglhm

@Chex2 said: Very worrying the fire suppression either failed or was not sufficient.

Saw somewhere that halogen suppression systems aren't allowed or something?

CConner

@Falzo said:
wonder what's there to actually burn in these tin cans... with what did they isolate?

if you look at this vid don't get your hopes to have anything from "SBG2" recovered at all:

https://twitter.com/_EstebanL/status/1369540663876980737

Yeah no wonder why they recommended everyone to activate their business recovery plan

Chex2

@sdglhm said:

@Chex2 said: Very worrying the fire suppression either failed or was not sufficient.

Saw somewhere that halogen suppression systems aren't allowed or something?

Yeah, concerns about the ozone layer I think.

netswitch

Argonite / IG55 is completely legal in EU and largely deployed in professional datacenters.
I am looking for information about fire suppression system at OVH but cannot find any.

gapper

I have a server in BHS1 and a backup on BHS2. Could I be affected like this in the future or are those DCs very far away from each other?

rick2610

update twitter:

Update 7:20am
Fire is over. Firefighters continue to cool the buildings with the water.
We don’t have the access to the site. That is why SBG1, SBG3, SBG4 won’t be restarted today.

CConner

@netswitch said:
Argonite / IG55 is completely legal in EU and largely deployed in professional datacenters.
I am looking for information about fire suppression system at OVH but cannot find any.

Whatever it is, one thing is safe to say, it wasn't very effective

Chex2

@gapper said:
I have a server in BHS1 and a backup on BHS2. Could I be affected like this in the future or are those DCs very far away from each other?

I think even if they're 10 miles apart you should have backups with different providers in different locations.