Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


GitLab employee ran rm -rf on production
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
«1

Comments

  • trewqtrewq Administrator, Patron Provider

    Everyone makes mistakes, he did in the wrong terminal. I've been working around this all day, times like this is when git shines.

  • Looooool

    It can happen to the best of us.

  • I'm actually quite surprised by a detailed incident report like this. Don't see this everyday.

    Thanked by 3Falzo Yura NanoG6
  • trewqtrewq Administrator, Patron Provider

    @busbr said:
    I'm actually quite surprised by a detailed incident report like this. Don't see this everyday.

    Especially while it's an ongoing issue. They're handling this like champs.

  • "Wait I.. FUCK!"

  • probably worst thing about that story:

    So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.

    should remind everyone to properly test any backup and restore process they may have in place...

  • trewqtrewq Administrator, Patron Provider

    @Falzo said:
    probably worst thing about that story:

    So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.

    should remind everyone to properly test any backup and restore process they may have in place...

    http://checkyourbackups.work

  • Yes this is his own fault, but, just curious:

    Why won't Linux OSs ask you to confirm (press y/n) after you type rm -rf...

    Windows would ask for your confirmation before removing files...

  • XIAOSpider97 said: Why won't Linux OSs ask you to confirm (press y/n) after you type rm -rf...

    because you explicitly told rm not to do so by using -f aka force.

    ( http://man7.org/linux/man-pages/man1/rm.1.html )

  • AnthonySmithAnthonySmith Member, Patron Provider
    edited February 2017

    XIAOSpider97 said: Why won't Linux OSs ask you to confirm

    Nope, some people modify rm to force this and output certain system specific info and request input, its good practice I suppose.

    edit: Sorry I read your questions as 'Dont Linix OS's ask', its early sorry.

  • @XIAOSpider97 said:
    Yes this is his own fault, but, just curious:

    Why won't Linux OSs ask you to confirm (press y/n) after you type rm -rf...

    Windows would ask for your confirmation before removing files...

    rm -rf

  • dgprasetyadgprasetya Member
    edited February 2017

    @trewq said:
    Everyone makes mistakes, he did in the wrong terminal. I've been working around this all day, times like this is when git shines.

    need to show the full hostname in the bash prompt for all users by default (e.g., “db1.staging.gitlab.com” instead of just “db1”)

    rofl..

    @Falzo said:
    probably worst thing about that story:

    So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.

    should remind everyone to properly test any backup and restore process they may have in place...

    agree..

    also the bucket is empty... hahahaha

    i think gitlab should hire sysadmin or devops from lowendtalk activist ;)

  • maldoviamaldovia Member
    edited February 2017

    @XIAOSpider97 said:
    Yes this is his own fault, but, just curious:

    Why won't Linux OSs ask you to confirm (press y/n) after you type rm -rf...

    Windows would ask for your confirmation before removing files...

    It does actually, at least on Debian 8.7. I actually tested this last night. And rm -rf / wont really do anything, you must run rm -rf / --no-preserve-root. I'm not sure if this varies on CentOS / other distros.

    EDIT: this is wrong

  • mfsmfs Banned, Member
    edited February 2017

    XIAOSpider97 said: Why won't Linux OSs ask you to confirm (press y/n) after you type rm -rf...

    Well, because you already typed the -f switch; if rm were designed with a "Are you sure? Give it another read, you absolute mad man" message when the -r or -f switches are invoked, someone would then add a "-y" switch or similar to work around that nagging

    There are wrappers for rm but those give a fake sense of security; there's always someone proposing to add "safe nets" to work around bugs ( rm even bricked hardware featuring poor EFI implementations) or human faults. Someone designed softwares to (attempt to) showcase what a command would do before running it (without actually really sandboxing the code, so leading to potential unpredicted results - even worse if you're running it as root) but I still think that you should rather keep reminding people what root means

    If you're running the command as root there's no ultimate babysitting that could keep you safe. Windows, if any, helped (at least in the past) in developing bad practices like "run everything always as administrator, there will always be some safety net for you (or not? ooops)"

    maldovia said: --no-preserve-root

    that protects you only from the most obnoxious example of "rm -rf", it's been around for a while

  • maldoviamaldovia Member
    edited February 2017

    @mfs said:

    XIAOSpider97 said: Why won't Linux OSs ask you to confirm (press y/n) after you type rm -rf...

    Well, because you already typed the -f switch; if rm were designed with a "Are you sure? Give it another read, you absolute mad man" message when the -r or -f switches are invoked, someone would then add a "-y" switch or similar to work around that nagging

    There are wrappers for rm but those give a fake sense of security; there's always someone proposing to add "safe nets" to work around bugs ( rm even bricked hardware featuring poor EFI implementations) or human faults. Someone designed softwares to (attempt to) showcase what a command would do before running it (without actually really sandboxing the code, so leading to potential unpredicted results - even worse if you're running it as root) but I still think that you should rather keep reminding people what root means

    If you're running the command as root there's no ultimate babysitting that could keep you safe. Windows, if any, helped (at least in the past) in developing bad practices like "run everything always as administrator, there will always be some safety net for you (or not? ooops)"

    maldovia said: --no-preserve-root

    that protects you only from the most obnoxious example of "rm -rf", it's been around for a while

    I went back and tested this on two separate DigitalOcean droplets.

    Debian 8.7: rm -rf / does nothing, rm -rf / --no-preserve-root will delete everything.

    Ubuntu 16.04: same as Debian 8.7

    I am not sure why the VM I tested this on last night asked me "Are you sure".

  • mfsmfs Banned, Member
    edited February 2017

    There are many other cases of rm -rf, --no-preserve-root won't stop you if you're using relative paths and/or wildcards. I doubt that the gitlab employee issued a "rm -rf /" to start with (I doubt you have any legitimate use case for it)

    EDIT

    @maldovia said:
    I am not sure why the VM I tested this on last night asked me "Are you sure".

    zsh?

  • jarjar Patron Provider, Top Host, Veteran

    Ouch. Poor admin, not gonna live this down for a while. Mistakes happen, no matter how much you do to avoid them. It's all about them backups, and making sure you at least don't make a mistake in how you backup ;)

  • maldovia said: I am not sure why the VM I tested this on last night asked me "Are you sure".

    There may be an alias already defined. Use \rm to bypass.

  • vfusevfuse Member, Host Rep

    They have a livestream on youtube now as well :)

    https://www.youtube.com/c/Gitlab/live

  • jarland said: making sure you at least don't make a mistake in how you backup ;)

    Yeah, that's what's weird that such a project had shitty backup plans....

  • YuraYura Member
    edited February 2017

    @datanoise said:

    jarland said: making sure you at least don't make a mistake in how you backup ;)

    Yeah, that's what's weird that such a project had shitty backup plans....

    If only they had some sort of version control and CI system in place...

    Hm...

  • The transparency is awesome, especially with that live stream

  • ...a document with the most detailled discussions and thoughts in it...
    ...a video stream... wow.
    This is by far the most transparent handling of such a case I've EVER seen.

  • OnApp_TerryOnApp_Terry Member
    edited February 2017

    Thought the discussion about their infrastructure was interesting. Sounded like 7 frontend servers using HAProxy. 2 Postgresql database servers, and multiple NFS mounts, all on Azure infrastructure. Also sounds like they're using a mix of the 'general purpose' and 'high-performance' machines for their dev & production environments.

    Thanked by 1datanoise
  • @southy said:
    ...a document with the most detailled discussions and thoughts in it...
    ...a video stream... wow.
    This is by far the most transparent handling of such a case I've EVER seen.

    the whole case draws a lot of attention. so this probably is the easiest and best way to keep damage to your reputation as low as possible and maybe even generate additional traction for the project itself from the sudden high level of interest...

  • I always use rm -rf on regular basis. I guess I am just living dangerously

  • crazy stuff.. prompted me to customise centmin mod's default PS1 command to display full hostname instead of short hostname now :)

    [18:02][[email protected] /home/nginx]#

  • Does not ask Linux if are you sure that you want to perform that action?

  • @WHT said:
    Does not ask Linux if are you sure that you want to perform that action?

    Generally rm is aliased to rm -i. The I is for interactive. Guess what that does. (Why are people seeming to be blaming the underlying OS for a stupid?)

  • WHT said: Does not ask Linux if are you sure that you want to perform that action?

    Pls see thread for the answer to that very question, at least 7 times.

Sign In or Register to comment.