Rethinking my backup system

dmmcintyre3

Right now, I simply rsync my files to 2 other systems and use http://sourceforge.net/projects/automysqlbackup/ for mysql then rsync that to 3 other systems. It works good, is reliable and I can easily restore the backup.

There is one issue with this, if I'm hacked the backups could be deleted/overwritten with garbage very easily. The main system can just ssh to the other servers and rm them or the files could be filled with junk which would be copied to the backup servers within 4 hours.

What should I do to improve this setup?

skagerrak

Create a specific backup-user. After the files have been transferred run a cronjob to copy the files into another user-account with respective privileges. If the main system gets corrupted you can simply delete the backup-user.

Infinity

I did think that that could happen (yep, thats an achievement for me, I'm pretty dopy and don't think things through), and which is why I was going to settle with archiving and dating (haha) the backups. And within the main archive is the SQL dumps and the www files and nsd zone files which gets transfered to one or more server.

[Notice how I said was going to settle, I still haven't set up backups! After at least 6 months of using VPS's for my hosting]

That will take up a lot of space though. I am open to lots more suggestions though :P Mine is probebly the least worthy of being used.

Francisco

Why not work in a pull basis and not a push?

You could just bind mount the data you want to pull RO to another folder then have an ssh user that you connect with with an sshkey.

If you keep snapshots from a couple days any hack wouldn't leave you screwed

Using a pull setup your backup server is safe from anyone breaking in, assuming they didn't get your private key/etc.

Francisco

Lm85H4gFkh3wk3

rsnapshot to the rescue! Too easy not to do it.

There's a rsnapshot howto section in one of the system setup guides listed in the LEB Wiki. Disclaimer : It's my guide. Disclaimer #2 : The guide needs some updates, but I don't know of any required updates to the rsnapshot section.

drmike

I'm cleaning out my pockets and came across a note mentioning mysqlhotcopy. This isn;t the thread I was thinking about but I know we've talked about mysql backups on a live site.

Does mysqlhotcopy work for doing something like that?

Just wanted to throw that out as a suggestion and see what folks thought about it.

sigh Another note down. Too many to go...

sleddog

rsync, then something like rdiff-backup on the backup server to create redundancy outside of the rsync directory.

Then, to protect your backup data from hacking in case your source server is rooted, use authprogs. Then the source server can only run the rsync command(s), and data in the rdiff-backup tree is safe.

kiloserve

dmmcintyre3 said: Right now, I simply rsync my files to 2 other systems and use

Stagger your backups, have the 2nd backup run every x number of days. Where X is a reasonable amount of days where you would know you've been hacked.

Auotmysqlbackup keeps daily/weekly/monthly backups so you're pretty safe there if you keep them all.

rm_

@dmmcintyre3
my cron job e-mails dumps of MySQL databases to a GMail account.
That way the sending machine has a "write-only" access to the backup store.
Sending files as attachments is easy to do with some program like 'sendEmail'; you may need to chop them into smaller pieces (use 'split') beforehand though.

rds100

You trust google/gmail with your database data? I hope you encrypt the dump before emailing it...

rm_

@rds100 Oh absolutely, forgot to mention that I also encrypt it with GPG before sending.

maxexcloo

rsnapshot is the way to go, simply amazing.

draco

maxexcloo said: rsnapshot is the way to go, simply amazing.

Do you use it from a remote server to pull the data, or back the data up locally (without rsyncing it elsewhere?)?

LowEndAdmin

@maxexcloo said: rsnapshot is the way to go, simply amazing

Unless you need to work on the medium and/or applications that don't really understand hardlinks.

JustinBoshoff

Must agree with Francisco on pull.
Check http://affa.sourceforge.net/
Works well, does hard links can automate reports via e-mail.
Works well for me.

maxexcloo

draco said: Do you use it from a remote server to pull the data, or back the data up locally (without rsyncing it elsewhere?)?

I pull data to two locations using restricted SSH Key access

draco

maxexcloo said: I pull data to two locations using restricted SSH Key access

If it's not too much to ask for, do you mind sharing if it's just 2 other low-end VPS or dedicated backup solutions (like rsync.net)?

eva2000

dmmcintyre3 said: Right now, I simply rsync my files to 2 other systems and use http://sourceforge.net/projects/automysqlbackup/ for mysql then rsync that to 3 other systems. It works good, is reliable and I can easily restore the backup.

There is one issue with this, if I'm hacked the backups could be deleted/overwritten with garbage very easily. The main system can just ssh to the other servers and rm them or the files could be filled with junk which would be copied to the backup servers within 4 hours.

What should I do to improve this setup?

I hear great things on WHT about http://www.bqbackup.com/ services maybe worth a try ? I'm writing my own backup script which can use a mix of rsync for data files along with mysqldump, mydumper, percona xtrabckup for mysql database backups. Still early stages.

But didn't really think of pull instead of pushing backups until this thread!

maxexcloo

draco said: If it's not too much to ask for, do you mind sharing if it's just 2 other low-end VPS or dedicated backup solutions (like rsync.net)?

One server is a Hostigation Backup VPS and the other is a local file server I have at home, feel free to ask for any info that could help you!

yomero

I like the @JustinBoshoff solution n_n Thinking on implementing one or 2 boxes for me

draco

maxexcloo said: One server is a Hostigation Backup VPS and the other is a local file server I have at home, feel free to ask for any info that could help you!

So for the Backup VPS, do you just do an "apt-get rsnapshot" and configure it to pull the data? What about databases? Do you just run mysqldump prior to rsnapshot running (maybe via a pre-rsnapshot script or a cronjob 10 minutes before) or what?

Am I right to assume that you don't have a local copy on the main machine? Right now my rsnapshot saves a local copy and then push (via rsync) to a rsync.net machine, followed by a pull to my home machine from rsync.net and I am thinking about getting rid of the local copy on the main machine to save space (and any crashes will render it useless anyway).

Oh, and what data do you backup, as I'm backing up /var, /home, /opt, /etc, /usr right now, anything else I should do?

Haha, sorry if I ask too much, I did read around but I thought I get these out of the system and ask for suggestions too.

LowEndAdmin

@eva2000 said: I hear great things on WHT about http://www.bqbackup.com/ services maybe worth a try?

I used to use bqbackup. And then moved to rsyncpalace. However what you get is just a shell account on a box with lots of disks, with severe restrictions and sometimes quite slow IO (as everyone is doing backups on those boxes). Not necessarily cheaper than a low end VPS either, where you get root access and can implement a lot fancier solutions.

dmmcintyre3

I ended up doing pull backups to another VPS using this script:

#!/bin/bash
cd /var/backup
automysqlbackup
rm backup.10 -rf
mv backup.9 backup.10
mv backup.8 backup.9
mv backup.7 backup.8
mv backup.6 backup.7
mv backup.5 backup.6
mv backup.4 backup.5
mv backup.3 backup.4
mv backup.2 backup.3
mv backup.1 backup.2
mv backup.0 backup.1
cp -R www backup.0
rsync -avz -e "ssh" root@tiger:/var/www/ /var/backup/www --exclude=*mysql/*

(I exclude /var/www/mysql to avoid re-backing up the mysql backups generated by the main VPS since I back up mysql again here.)

yomero

hmmm, logrotate?

sleddog

dmmcintyre3 said: rsync -avz -e "ssh" root@tiger:/var/www/ /var/backup/www --exclude=mysql/

If you backup server is exploited, can the hacker then access the webserver using the SSH keys?

eva2000

LowEndAdmin said: I used to use bqbackup. And then moved to rsyncpalace. However what you get is just a shell account on a box with lots of disks, with severe restrictions and sometimes quite slow IO (as everyone is doing backups on those boxes). Not necessarily cheaper than a low end VPS either, where you get root access and can implement a lot fancier solutions.

Ah didn't know that, thanks for the heads up!

maxexcloo

draco said: So for the Backup VPS, do you just do an "apt-get rsnapshot" and configure it to pull the data? What about databases? Do you just run mysqldump prior to rsnapshot running (maybe via a pre-rsnapshot script or a cronjob 10 minutes before) or what?

I install rsync and rsnapshot through the repos an create a config file in a user account (configured to pull data). I generate an SSH key and distribute that amongst the servers and then lock it to the backup servers IP and prevent it from using anything but rsync. I do not backup SQL (don't need to, yet).

draco said: Am I right to assume that you don't have a local copy on the main machine? Right now my rsnapshot saves a local copy and then push (via rsync) to a rsync.net machine, followed by a pull to my home machine from rsync.net and I am thinking about getting rid of the local copy on the main machine to save space (and any crashes will render it useless anyway).

I don't store any backup data on the backed up machines.

draco said: Oh, and what data do you backup, as I'm backing up /var, /home, /opt, /etc, /usr right now, anything else I should do?

I just backup /home/nameofuser