Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


GitLab employee ran rm -rf on production - Page 2
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

GitLab employee ran rm -rf on production

2»

Comments

  • raindog308raindog308 Administrator, Veteran

    I was a Unix admin for many years and despite making all the classic stupid mistakes, this particular one is one I never made. An grizzled old vet said "run ls with the same args first" and that advice has served me very well.

    However, I was once sitting next to a colleague in the 90s (an era of 2GB SCSI drives) who got up, got coffee, came back to his desk, and said "wow, this rm is still running..." That was a long night.

  • I've nuked a server like this before, but never at work.

  • Worst I ever did was wake up from going to the bar at lunch and reformatted the wrong "server" hostname thinking it was the ircd dev but actually was the production IRC server

  • @Falzo said:
    probably worst thing about that story:

    So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.

    should remind everyone to properly test any backup and restore process they may have in place...

    maybe a blessing in disguise to highlight this issue !

  • raindog308 said: got up, got coffee, came back to his desk, and said "wow, this rm is still running..."

    Thanked by 1Hxxx
  • Honestly this gave them a good PR. With all the "transparency" and live stream, details of steps and processes.

  • JustAMacUserJustAMacUser Member
    edited February 2017

    @raindog308 said:
    An grizzled old vet said "run ls with the same args first" and that advice has served me very well.

    ^ This.

    I've also gotten in the habit that if I have multiple terminal windows open, I make sure the production ones have a different background colour (usually a shade of red).

    ... That being said, while I've never misused rm, I have accidentally run chmod -R and broken things.

  • raindog308raindog308 Administrator, Veteran

    doughmanes said: production IRC server

    I just love that phrase.

    Thanked by 4Yura southy WSS Clouvider
  • eva2000eva2000 Veteran
    edited February 2017

    luckily in 17+ yrs never had made such a mistake and have routinely have 10-30 SSH windows open simultaneously heh

    for my Centmin Mod users just added a pwdh alias which displays hostname + pwd together just as an extra check before running commands https://community.centminmod.com/threads/add-pwdh-alias-command.10309/ :)

    cd /home/nginx
    pwdh
    host1.domain.com /home/nginx
    
  • emgemg Veteran

    The ancient VAX/VMS operating system keeps many versions of files as they change (edit/save cycles, for example). The VMS "purge" command deletes all the old versions except the most recent, current version. In VMS, the purge command is safe and relatively benign.

    After working with VAX/VMS for several years, my next project worked with a different real-time operating system, which also had a file versioning scheme. On that OS, the "purge" command means, "delete this directory and everything in it, and then do the same for all subdirectories recursively". On that OS, the purge command is effectively the same as "rm -rf *"

    (I vaguely remember the OS to be iRMX86, but a quick web search did not reveal the "purge" command. Perhaps it was a different OS. It was a long time ago.)

    Needless to say, my first day on that project was not a fun one, starting immediately after I typed the purge command. A more experienced co-worker and I spent the rest of the day restoring the deleted files and directories by manually editing the directory structures at a low level. It took the rest of the day to get the system back to the point where I had started that morning. After that incident, it took time to earn my credibility on that project. :-)

  • @JustAMacUser said:
    I've also gotten in the habit that if I have multiple terminal windows open, I make sure the production ones have a different background colour (usually a shade of red).

    I do this, too. Another good idea is to have separate desktops and/or accounts that you switch to only when you need those escalated privileges.

    ... That being said, while I've never misused rm, I have accidentally run chmod -R and broken things.

    I wonder if it might not be time to consider eliminating some of these "dangerous" commands from production servers (or the shells) completely. I mean, I remember a time when you actually would delete data from a database, because 1MB was precious space! Now we just flag rows as "deleted" or move it to a "deleted" table, because TBs of space is cheap. Maybe rm shouldn't even be available on certain servers.

  • @impossiblystupid said:

    @JustAMacUser said:
    I've also gotten in the habit that if I have multiple terminal windows open, I make sure the production ones have a different background colour (usually a shade of red).

    I do this, too. Another good idea is to have separate desktops and/or accounts that you switch to only when you need those escalated privileges.

    ... That being said, while I've never misused rm, I have accidentally run chmod -R and broken things.

    I wonder if it might not be time to consider eliminating some of these "dangerous" commands from production servers (or the shells) completely. I mean, I remember a time when you actually would delete data from a database, because 1MB was precious space! Now we just flag rows as "deleted" or move it to a "deleted" table, because TBs of space is cheap. Maybe rm shouldn't even be available on certain servers.

    I have it redirected on production and on my dev server. I moved it to something I could never mess up:

    rm --> del

    So instead of "rm -rf," it needs to be "del -rf" :D

    Thanked by 1Hxxx
  • @doghouch said:
    So instead of "rm -rf," it needs to be "del -rf" :D

    A typo can still wipe the wrong thing, though. I'm questioning the whole notion of destroying data in the first place. Just use mv to drop it into /tmp or some other location, just like it works for the Trash on a Mac or the Recycle Bin on Windows.

    If I really wanted the data gone, maybe I'd have to chroot to that directory to run a destructive command. Or reboot the machine to clean it out. Or spin up a brand new replacement VM configured to simply not have that data in the first place.

    Thanked by 1Hxxx
  • doghouchdoghouch Member
    edited February 2017

    @impossiblystupid said:

    @doghouch said:
    So instead of "rm -rf," it needs to be "del -rf" :D

    A typo can still wipe the wrong thing, though. I'm questioning the whole notion of destroying data in the first place. Just use mv to drop it into /tmp or some other location, just like it works for the Trash on a Mac or the Recycle Bin on Windows.

    If I really wanted the data gone, maybe I'd have to chroot to that directory to run a destructive command. Or reboot the machine to clean it out. Or spin up a brand new replacement VM configured to simply not have that data in the first place.

    Well, only two of my servers have this, so I know when I'm on the wrong server. I also have it ask me "Do you really want to proceed? CTRL+C to cancel operation"

  • It can happen to anyone...but on the subject of backups, they seriously didn't test any of their backup systems?!

  • They seem to be friendly people, honestly, I might buy . lol

  • I've rm -rf * the wrong directory before on a cpanel machine and killed a website.

    Thankfully just the site content and not the DB and I had a backup from the day before so was able to restore them

Sign In or Register to comment.