New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Best backup "strategy"
Hi,
I currently have 3 backupservers:
All of my devices (servers, computers and my mobile) put their important files on server1. Then every night server1 syncs (over rsync) all files with server2. Futhermore, I manually sync server1 with server3 over rsync every few days. I think my files are safe now, but when a disk on server1 corrupts and destroys a few files, these destroyed files will be synced with server2 and I might not notice that as the servers sync automatically. I wonder if there is a better way of how to use these 3 servers as backupservers. The files on this servers are all of my private files (pictures, my music, my documents etc.) so they are really important to me.
Thank you!
Thanked by 1aglodek
Comments
I personally use a remote backup solution for my personal files such as LiveDrive or BackBlaze. I realise however that these aren't popular among many as you're pretty much handing your data over to whatever authority wants to pry in them.
I'd recommend storing some snapshot like TARs (TAR files of your backups weekly and monthly etc) on server3 if it has the space to accommodate such files.
Backup retention will protect you from this.
Have a backup per day save in a different folder then if one gets corrupted you still have the days before, I have 7 days retention on my main backup folder then on my PSN I have 30 days retention
Yeah, would be an idea. I already thought about doing the backup only every 2 days.
Anyway, even when I save let's say a backup for 3 days, when my provider's disk corrupts in 3 days and they don't notice that. I'll think everything is fine and will delete my backup
Thanks for suggesting external backup providers. But I already bought these 3 servers and want to use them :P
use something like rdiffbackup or attic to do this step, so you have some history in changes and be able to restore older versions too...
First thing I'd do is address this corruption issue. It shouldn't be happening.
Here's my backup setup:
I like that plan, but I'm very paranoid about keeping truly personal data on servers I don't have complete physical control over. </tinfoil hat>
I agree, the only server that I keep backups on that I don't own is CrashPlan+, but it allows you to set your own encryption key. The encryption is 448-bit with a custom 448-bit key (more info here).
So many nude picture, no? Lol j/k
why not use encfs and rsync? cron also helps.
Duply is great for that. It allows you to use your own servers, specify backup retention times and encrypted backups.
I have a similar setup but also do date based fragmenting of the files e.g.
/backup/day1/device1
/backup/day2/device1
rsyncd to a vps then replicate around the world. Never had a corruption issue.
Local machine - lan NAS - was (samba vpn tunnel) - replicated to was2,was3, etc.
From laptop to 3 drives on my home server (daily, weekly, monthly on each drive with no raid) using cronjob, and a copy of just my documents and important files (saved games) is synced to seafile on my dedicated server instantly. Most my photos are on my home server and on cold storage such as DVD's, I have never had a DVD/CD fail on me yet!
Stuff like config files, website data and non critical MySQL is backed up to ramnode vps and kimsufi VPS every week, and all MySQL is backed up every hour to my home server's 4th HDD and Kimsufi. I can run a live copy of my dedi (znc/mysql/websites/seafile) from my kimsufi if needed.
Every few months or after OS reinstall I also do full disk image of my main laptops with DD to an external HDD, the plan being I can restore the image if my HDD dies or I install too much crap, seafile will sync my documents back and I can manually restore anything else missing from my home server or DVD's. This setup has worked fine for many years and many different PC's, I don't see corruption as being a big issue.
You can use crashplan software for free to backup your devices between them (not using crashplan storage)
Take a look at it. I do this way with my computers and my storage vps, it maintains lots of versions and the storage and transfer is deduped, so it is very efficient.
Not sure if this is the solution you're looking for but you can also use apps like dropbox.com or backupify and the like to store your important files and it syncs automatically as well to all your devices and you don't have to worry anymore. Not affiliated with them though but I use them.
Some sort of backup rotation scheme - http://en.wikipedia.org/wiki/Backup_rotation_scheme.
Grandfather/father/son works well - eg you make daily backups and keep each backup for a week. One of those backups gets relabelled as your weekly backup and you keep those for a month. One of those weekly backups gets relabelled as your monthly backup & you keep that for a quarter. Combines the benefit of frequent backups with the benefit of being able to go back several months if there's a corruption.
Also:
be sure some of the backups are offsite in case the server dies completely (I guess you've got that covered)
be sure you're buying backup servers from different companies (in case someone vanishes overnight) and in different locations (in case of natural disaster).
Thank you for all your suggestions!
As said previously, I don't want to use a 3rd party service like Dropbox/MEGA and so on. I would like to use my own servers.
I changed my backup behaviour: I use a grandfather/father/son system now :P
server1 syncs everyday to server2
server1 syncs every 3 day to a directory on server 3
server1 syncs every 7 day to another directory on server 3
I think this is sufficient now, isn't it?
The following, I consider one of the best I've ever heard/read. I leave it t your own judgement.
http://www.howtoforge.com/forums/showthread.php?t=61008 ...tahunasky ... 27th April 2013, 03:18
I do backups every weekday, and then a weekly backup for 4 weeks, that way i only ever have 8 backup files, but my backups go back a month, and i can even do a monthly backup if needed. Everything gets over-written, so no need to delete anything. and the weekly backups are stored off-site.
eg:
backup-daily-monday.tar to backup-daily-thursday.tar, backup-weekly-01.tar to backup-weekly-04.tar
I also run a script on a backup server, which has a wake-on-bios function. Every morning at 1am the server turns on, runs through the list of other servers to check if the servers are running, if so syncs with it/them (emails/websites/cloud storage area, and copies over the backup file from that night, and turns itself off... if it finds a server that is down, it will configure itself to be the server that is down (changing IP/hostname etc) then emails me to say what has happened. Then i just have to send my mum or sister an email to swap out the downed server with a pre-configured spare or they can swap out a faulty HD with pre-configured spares. I can then start the replacement server (with wake-on-lan) when i am ready and configure remotely, and restore files etc from the backup server which does the syncing.
By doing this i have only had to do one restore from backup in the last couple of years, and is particularly useful for me as i spend alot of the year traveling and living in other countries, and can be away for home for 9 months or more at a time.
But what is the best place to put your data on? VPS (OVZ/KVM/XEN-VM...) or Dedi? or some Cloud Hosting? I would like to know your opinions.
I use my raspberry pi with a usb laptop drive. Lets me backup all I want over the lan xD But when I'm away I use duplicity and send the data to a ramnode vps or dedi.
I'm still blown away that the guy's sister and mother know how to swap out servers/HDDs like it's no big deal! With those around me, it's been a big challenge to even teach them how to plug in and copy files to a USB key. I've been working on them for YEARS. No joke.
I do:
So come restore time I can:
+1 for doing backup-rotation with rdiff-backup
In general the best backup strategy is to make a backup of the backup (in a remote location).
There are many way to apply this as already posted on this thread you can use a NAS or an external disk or a hosted solution or a mix of all these together ,
I use external backup drive mainly from WD that supports password to access to their USB disk.
I also use a VPS server from DigitalOcean
i wrote an article on this which may help you out https://rackerjack.uk/backup/
One of the best threads of 2015 so far, IMHO.
Cheers
Its from 2014 -.-
Yes, and it is like the gift that keeps of giving ... best of 2015 (so far) too, lol
Edit: And I forgot to add, I just use simple stuff for my needs: