#### Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

# GitLab employee ran rm -rf on production

Member

Everyone makes mistakes, he did in the wrong terminal. I've been working around this all day, times like this is when git shines.

• Member

Looooool

It can happen to the best of us.

• Member

I'm actually quite surprised by a detailed incident report like this. Don't see this everyday.

Thanked by 3Falzo Yura NanoG6

@busbr said:
I'm actually quite surprised by a detailed incident report like this. Don't see this everyday.

Especially while it's an ongoing issue. They're handling this like champs.

• Member

"Wait I.. FUCK!"

I won't be back until @bsdguy is released.

• Member

probably worst thing about that story:

So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.

should remind everyone to properly test any backup and restore process they may have in place...

most recommended Provider: First-Root KVM Power-Edition /w SSD
UltraVPS.eu KVM in US/UK/NL/DE: 15% off first 6 month | Netcup VPS/rootDS - 5€ off: 36nc15279180197 (ref)

@Falzo said:
probably worst thing about that story:

So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.

should remind everyone to properly test any backup and restore process they may have in place...

http://checkyourbackups.work

• Member, Provider

Yes this is his own fault, but, just curious:

Why won't Linux OSs ask you to confirm (press y/n) after you type rm -rf...

Pump Cloud has been sold on Sep 1 2018.

• Member

XIAOSpider97 said: Why won't Linux OSs ask you to confirm (press y/n) after you type rm -rf...

because you explicitly told rm not to do so by using -f aka force.

most recommended Provider: First-Root KVM Power-Edition /w SSD
UltraVPS.eu KVM in US/UK/NL/DE: 15% off first 6 month | Netcup VPS/rootDS - 5€ off: 36nc15279180197 (ref)

• Top Provider
edited February 2017

XIAOSpider97 said: Why won't Linux OSs ask you to confirm

Nope, some people modify rm to force this and output certain system specific info and request input, its good practice I suppose.

Had enough of the scams on lowendbox, lowendtalk is now being infiltrated by corruption so I have chosen to make an low end exit #lexit for now - you can find me HERE

• Member

@XIAOSpider97 said:
Yes this is his own fault, but, just curious:

Why won't Linux OSs ask you to confirm (press y/n) after you type rm -rf...

rm -rf

• Member
edited February 2017

@trewq said:
Everyone makes mistakes, he did in the wrong terminal. I've been working around this all day, times like this is when git shines.

need to show the full hostname in the bash prompt for all users by default (e.g., “db1.staging.gitlab.com” instead of just “db1”)

rofl..

@Falzo said:
probably worst thing about that story:

So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.

should remind everyone to properly test any backup and restore process they may have in place...

agree..

also the bucket is empty... hahahaha

i think gitlab should hire sysadmin or devops from lowendtalk activist

"The quieter you become, the more you are able to listen" dgprasetya.com

• Member
edited February 2017

@XIAOSpider97 said:
Yes this is his own fault, but, just curious:

Why won't Linux OSs ask you to confirm (press y/n) after you type rm -rf...

It does actually, at least on Debian 8.7. I actually tested this last night. And rm -rf / wont really do anything, you must run rm -rf / --no-preserve-root. I'm not sure if this varies on CentOS / other distros.

EDIT: this is wrong

thanked by nekki sister !

• Banned, Member
edited February 2017

XIAOSpider97 said: Why won't Linux OSs ask you to confirm (press y/n) after you type rm -rf...

Well, because you already typed the -f switch; if rm were designed with a "Are you sure? Give it another read, you absolute mad man" message when the -r or -f switches are invoked, someone would then add a "-y" switch or similar to work around that nagging

There are wrappers for rm but those give a fake sense of security; there's always someone proposing to add "safe nets" to work around bugs ( rm even bricked hardware featuring poor EFI implementations) or human faults. Someone designed softwares to (attempt to) showcase what a command would do before running it (without actually really sandboxing the code, so leading to potential unpredicted results - even worse if you're running it as root) but I still think that you should rather keep reminding people what root means

If you're running the command as root there's no ultimate babysitting that could keep you safe. Windows, if any, helped (at least in the past) in developing bad practices like "run everything always as administrator, there will always be some safety net for you (or not? ooops)"

maldovia said: --no-preserve-root

that protects you only from the most obnoxious example of "rm -rf", it's been around for a while

I've left LET, account made inactive on request.

• Member
edited February 2017

@mfs said:

XIAOSpider97 said: Why won't Linux OSs ask you to confirm (press y/n) after you type rm -rf...

Well, because you already typed the -f switch; if rm were designed with a "Are you sure? Give it another read, you absolute mad man" message when the -r or -f switches are invoked, someone would then add a "-y" switch or similar to work around that nagging

There are wrappers for rm but those give a fake sense of security; there's always someone proposing to add "safe nets" to work around bugs ( rm even bricked hardware featuring poor EFI implementations) or human faults. Someone designed softwares to (attempt to) showcase what a command would do before running it (without actually really sandboxing the code, so leading to potential unpredicted results - even worse if you're running it as root) but I still think that you should rather keep reminding people what root means

If you're running the command as root there's no ultimate babysitting that could keep you safe. Windows, if any, helped (at least in the past) in developing bad practices like "run everything always as administrator, there will always be some safety net for you (or not? ooops)"

maldovia said: --no-preserve-root

that protects you only from the most obnoxious example of "rm -rf", it's been around for a while

I went back and tested this on two separate DigitalOcean droplets.

Debian 8.7: rm -rf / does nothing, rm -rf / --no-preserve-root will delete everything.

Ubuntu 16.04: same as Debian 8.7

I am not sure why the VM I tested this on last night asked me "Are you sure".

thanked by nekki sister !

• Banned, Member
edited February 2017

There are many other cases of rm -rf, --no-preserve-root won't stop you if you're using relative paths and/or wildcards. I doubt that the gitlab employee issued a "rm -rf /" to start with (I doubt you have any legitimate use case for it)

EDIT

@maldovia said:
I am not sure why the VM I tested this on last night asked me "Are you sure".

zsh?

I've left LET, account made inactive on request.

• Provider

Ouch. Poor admin, not gonna live this down for a while. Mistakes happen, no matter how much you do to avoid them. It's all about them backups, and making sure you at least don't make a mistake in how you backup

• Member

maldovia said: I am not sure why the VM I tested this on last night asked me "Are you sure".

There may be an alias already defined. Use \rm to bypass.

183 VPS tested in real time and counting... (86 active LEB providers).
All links I'm posting are tracked affiliate links, whenever the provider has an affiliate program.
• Member, Provider

They have a livestream on youtube now as well

NIXStats monitoring Web, Server(Linux, Windows - $6.95/m), Logging (Free!) and Blacklists (start at 512 for$3.75/m) - Uptime Report - API Docs

• Member

jarland said: making sure you at least don't make a mistake in how you backup

Yeah, that's what's weird that such a project had shitty backup plans....

don't buy what you don't need: you'll save money and will end up able to grab a quality VPS when really needed.

• Member
edited February 2017

@datanoise said:

jarland said: making sure you at least don't make a mistake in how you backup

Yeah, that's what's weird that such a project had shitty backup plans....

If only they had some sort of version control and CI system in place...

Hm...

• Member, Provider

The transparency is awesome, especially with that live stream

ik moet poepen

• Member

...a document with the most detailled discussions and thoughts in it...
...a video stream... wow.
This is by far the most transparent handling of such a case I've EVER seen.

• Member, Provider
edited February 2017

Thought the discussion about their infrastructure was interesting. Sounded like 7 frontend servers using HAProxy. 2 Postgresql database servers, and multiple NFS mounts, all on Azure infrastructure. Also sounds like they're using a mix of the 'general purpose' and 'high-performance' machines for their dev & production environments.

Thanked by 1datanoise

OnApp + CDN.net
[email protected]

• Member

@southy said:
...a document with the most detailled discussions and thoughts in it...
...a video stream... wow.
This is by far the most transparent handling of such a case I've EVER seen.

the whole case draws a lot of attention. so this probably is the easiest and best way to keep damage to your reputation as low as possible and maybe even generate additional traction for the project itself from the sudden high level of interest...

most recommended Provider: First-Root KVM Power-Edition /w SSD
UltraVPS.eu KVM in US/UK/NL/DE: 15% off first 6 month | Netcup VPS/rootDS - 5€ off: 36nc15279180197 (ref)

• Member

I always use rm -rf on regular basis. I guess I am just living dangerously

• Member

crazy stuff.. prompted me to customise centmin mod's default PS1 command to display full hostname instead of short hostname now

[18:02][[email protected] /home/nginx]#

* Centmin Mod Project (HTTP/2 support + ngx_pagespeed + Nginx Lua + Vhost Stats)
* Centmin Mod LEMP Stack Quick Install Guide
• Member

Does not ask Linux if are you sure that you want to perform that action?

• Member

@WHT said:
Does not ask Linux if are you sure that you want to perform that action?

Generally rm is aliased to rm -i. The I is for interactive. Guess what that does. (Why are people seeming to be blaming the underlying OS for a stupid?)

I won't be back until @bsdguy is released.

• Moderator

WHT said: Does not ask Linux if are you sure that you want to perform that action?

Pls see thread for the answer to that very question, at least 7 times.

Here lies Nekki. He loved massive amounts of storage, K-Pop and calling people cunts.

• Moderator

I was a Unix admin for many years and despite making all the classic stupid mistakes, this particular one is one I never made. An grizzled old vet said "run ls with the same args first" and that advice has served me very well.

However, I was once sitting next to a colleague in the 90s (an era of 2GB SCSI drives) who got up, got coffee, came back to his desk, and said "wow, this rm is still running..." That was a long night.

For LET support, please visit the interim support desk.

• Moderator

I've nuked a server like this before, but never at work.

Here lies Nekki. He loved massive amounts of storage, K-Pop and calling people cunts.

• Member

Worst I ever did was wake up from going to the bar at lunch and reformatted the wrong "server" hostname thinking it was the ircd dev but actually was the production IRC server

How to clean up a questionable reputation: throw the kids some BF/CM offers.

• Member

@Falzo said:
probably worst thing about that story:

So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.

should remind everyone to properly test any backup and restore process they may have in place...

maybe a blessing in disguise to highlight this issue !

* Centmin Mod Project (HTTP/2 support + ngx_pagespeed + Nginx Lua + Vhost Stats)
* Centmin Mod LEMP Stack Quick Install Guide
• Member

raindog308 said: got up, got coffee, came back to his desk, and said "wow, this rm is still running..."

Thanked by 1Hxxx

I am here occasionally nowadays, because I really prefer https://talk.lowendspirit.com . You should try it, it is fat-free, delicious with fresh ingredients combined with the deep knowledge of the old chefs!

• Member

Honestly this gave them a good PR. With all the "transparency" and live stream, details of steps and processes.

• Member
edited February 2017

@raindog308 said:
An grizzled old vet said "run ls with the same args first" and that advice has served me very well.

^ This.

I've also gotten in the habit that if I have multiple terminal windows open, I make sure the production ones have a different background colour (usually a shade of red).

... That being said, while I've never misused rm, I have accidentally run chmod -R and broken things.

• Moderator

doughmanes said: production IRC server

I just love that phrase.

Thanked by 4Yura southy WSS Clouvider

For LET support, please visit the interim support desk.

• Member
edited February 2017

luckily in 17+ yrs never had made such a mistake and have routinely have 10-30 SSH windows open simultaneously heh

for my Centmin Mod users just added a pwdh alias which displays hostname + pwd together just as an extra check before running commands https://community.centminmod.com/threads/add-pwdh-alias-command.10309/

cd /home/nginx
pwdh
host1.domain.com /home/nginx

* Centmin Mod Project (HTTP/2 support + ngx_pagespeed + Nginx Lua + Vhost Stats)
* Centmin Mod LEMP Stack Quick Install Guide
• Member

The ancient VAX/VMS operating system keeps many versions of files as they change (edit/save cycles, for example). The VMS "purge" command deletes all the old versions except the most recent, current version. In VMS, the purge command is safe and relatively benign.

After working with VAX/VMS for several years, my next project worked with a different real-time operating system, which also had a file versioning scheme. On that OS, the "purge" command means, "delete this directory and everything in it, and then do the same for all subdirectories recursively". On that OS, the purge command is effectively the same as "rm -rf *"

(I vaguely remember the OS to be iRMX86, but a quick web search did not reveal the "purge" command. Perhaps it was a different OS. It was a long time ago.)

Needless to say, my first day on that project was not a fun one, starting immediately after I typed the purge command. A more experienced co-worker and I spent the rest of the day restoring the deleted files and directories by manually editing the directory structures at a low level. It took the rest of the day to get the system back to the point where I had started that morning. After that incident, it took time to earn my credibility on that project. :-)

• Member

@JustAMacUser said:
I've also gotten in the habit that if I have multiple terminal windows open, I make sure the production ones have a different background colour (usually a shade of red).

I do this, too. Another good idea is to have separate desktops and/or accounts that you switch to only when you need those escalated privileges.

... That being said, while I've never misused rm, I have accidentally run chmod -R and broken things.

I wonder if it might not be time to consider eliminating some of these "dangerous" commands from production servers (or the shells) completely. I mean, I remember a time when you actually would delete data from a database, because 1MB was precious space! Now we just flag rows as "deleted" or move it to a "deleted" table, because TBs of space is cheap. Maybe rm shouldn't even be available on certain servers.

I am Impossibly Stupid. Hailed by @jarland as an "incessantly belligerent buffoon." Available for parties. Book early to avoid disappointment.

• Member

@impossiblystupid said:

@JustAMacUser said:
I've also gotten in the habit that if I have multiple terminal windows open, I make sure the production ones have a different background colour (usually a shade of red).

I do this, too. Another good idea is to have separate desktops and/or accounts that you switch to only when you need those escalated privileges.

... That being said, while I've never misused rm, I have accidentally run chmod -R and broken things.

I wonder if it might not be time to consider eliminating some of these "dangerous" commands from production servers (or the shells) completely. I mean, I remember a time when you actually would delete data from a database, because 1MB was precious space! Now we just flag rows as "deleted" or move it to a "deleted" table, because TBs of space is cheap. Maybe rm shouldn't even be available on certain servers.

I have it redirected on production and on my dev server. I moved it to something I could never mess up:

rm --> del

So instead of "rm -rf," it needs to be "del -rf"

Thanked by 1Hxxx
• Member

@doghouch said:
So instead of "rm -rf," it needs to be "del -rf"

A typo can still wipe the wrong thing, though. I'm questioning the whole notion of destroying data in the first place. Just use mv to drop it into /tmp or some other location, just like it works for the Trash on a Mac or the Recycle Bin on Windows.

If I really wanted the data gone, maybe I'd have to chroot to that directory to run a destructive command. Or reboot the machine to clean it out. Or spin up a brand new replacement VM configured to simply not have that data in the first place.

Thanked by 1Hxxx

I am Impossibly Stupid. Hailed by @jarland as an "incessantly belligerent buffoon." Available for parties. Book early to avoid disappointment.

• Member
edited February 2017

@impossiblystupid said:

@doghouch said:
So instead of "rm -rf," it needs to be "del -rf"

A typo can still wipe the wrong thing, though. I'm questioning the whole notion of destroying data in the first place. Just use mv to drop it into /tmp or some other location, just like it works for the Trash on a Mac or the Recycle Bin on Windows.

If I really wanted the data gone, maybe I'd have to chroot to that directory to run a destructive command. Or reboot the machine to clean it out. Or spin up a brand new replacement VM configured to simply not have that data in the first place.

Well, only two of my servers have this, so I know when I'm on the wrong server. I also have it ask me "Do you really want to proceed? CTRL+C to cancel operation"

• Member

It can happen to anyone...but on the subject of backups, they seriously didn't test any of their backup systems?!

• Member

They seem to be friendly people, honestly, I might buy . lol

• Member

I've rm -rf * the wrong directory before on a cpanel machine and killed a website.

Thankfully just the site content and not the DB and I had a backup from the day before so was able to restore them