Thoughts about really cheap but really good backup
A current topic in the Cest Pit reminded me to do some calculations. It's about a quite attractive storage server offer from ServDiscount that comes with 4 x 6 TB disks plus a small SSD (for the OS, etc.).
Now, front-up, if you happen to host Terabytes of say movies or porn or whatever that needs to be online then this post isn't for you, sorry, you'll just need to continue paying for large storage/backup (or run a high risk operation).
Many of us, though (actually most I guess) just run small to mid size web sites, mail servers, etc. that is, servers needing a couple of 10 or at most a couple of 100 GB in terms of space. My most busy web server for example, a mid size quite busy community, runs on a VPS with 4 cores, 4 GB memory and a 100 GB SSD. When I say that my community is quite busy I mean that there are some tens of MB of new content (real content not images) per day. Wait! "some ten MBs per day?"? That's peanuts!
Yes, that's peanuts but also what probably 80+% of VPS admins actually have. Now, assume that I had ten of those communities; then I'd be in the "couple of 100s MB per day", still a very easily managable amount of data to backup.
Yet I, like many others, had GBs/day in backup data due to bad planning and managing and I once had a 400 GB storage/backup server overflowing.
So, point #1 is: Do not simply backup everything, not even everything "changed" in the eyes of a non smart backup utility. Get some understanding and have a smart plan of what really needs to be backed up (and how frequently). Mysqldump and the like are your friends ...
Next point: Yes that fat storage server I mentioned above is a dangerously attractive offer, no doubts. But there's an ugly "but". It (like virtually all cheap storage servers) comes with 2nd hand or used before hard drives. Plus, usually those drives haven't been chosen for quality but for price. "No problem" I hear you say "I'm putting them in a Raid 1/5/6/Z anyway". Well, yes and no. "I run a Raid6 set" is quite OK when at the very minimum the disks are new or well known since "birth"; preferably I'd like to have and know a good Raid controller too, plus I'd like to have its sibling waiting as a replacement in my stock. Then, yes, it's indeed reasonable to sleep well and to think that the chances of 2 (or 3 with the higher end Raid schemes) drives failing at the same point in time is very, very low.
If however your drives are 2nd hand and you know nothing about them (like how have they been used?) and your machines Raid controller is called, well "Raid controller" (as opposed to brand/model/fw) then chances of multiple disks failing - or a staggered fail - are not at all that low.
Plus, keep that in mind as it's an important point that may turn against you, we are talking about dedicated servers here. Let me translate that clearly: It's - at least mainly - your problem if everything works fine (after the initial phase, say a week or so). Sure, you have a right to get a drive replacement if a drive fails but that's still just another lottery in terms of what (and how quickly) you get for a replacement. With a VPS it's the providers problem, not yours.
Plus: Raid recovery can take a loooong time. Well noted, those hours (or days) are very sensitive because if another drive fails during recovery things quickly get seriously ugly. To make it a bit funnier, guess what's about the most stressful time in a disks life? Right, during recovery, which means that the risk of another drive failing during recovery is not average; it's particularly high.
And no, I'm not exaggerating. Just a while ago a good and reputable provider lost all data of all VMs on a complete node due to disk and/or Raid controller problems. And those guys really know their job, but still they couldn't recover the data.
Unfortunately, the alternative is to either rent a very expensive server with expressly new disks, preferably with a very reputable provider or to buy such a server yourself and to colocate it.
Which brings me back to the beginning. Do we really need to play that ugly game?
I'm playing with (and have begun actually testing) an alternative many might first laugh at. I am providing backup. As in "at home/in my office". Hear me out.
A 2 TB disk, brand spanking new and good quality costs about 100 $/€. At this point another (maybe) shrewdness of mine enters the game. I hate wasting energy. So I use a <= 10W Alix based box, which happens to have an M2 slot (so I have put a nice 120 GB SSD on the board), even a SATA slot (which I ignore) and a USB 3 slot - which pretty much shouted "use me!". Next to it I have a dual external disks enclosure connected via USB 3 so the OS on the Alix router box sees two 2TB drives and happily Raids them as a R1 set.
Let's calculate: The drives plus the enclosure cost me about 200€ all together. The alix router isn't counted because I had it anyway. And btw it has 4 AMD (low power) cores and plenty RAM (4 GB) so the additional load is pretty insignificant. Calculated over just 1 year the cost is about 17€/mo. Not at all bad, being in the price range of a comparable 2 core Atom dedi with 1 or 2 TB disks. Considering a life time of a ridiculously low 3 years the costs are about 5.5€/mo. Try to beat that! And for that peanuts money I have new and good quality (in my case Toshiba) drives and full control of the machine!
Now comes the potential bottleneck, traffic. I have a (quite average I guess) DSL with 50 (or 100? don't know, don't care) down and 5 (or 10?) upstream. And my provider, like most based on what I hear, doesn't get icky as long as my volume doesn't go far beyond 250 or 300 GB/month. Let's say I allocate 60 GB of that to my backup solution. That translates to about 2 GB/day - which is far more than I - and I guess many if not most of us - actually need if we plan our backup properly.
Last but not least I have to mention a trick, a bit of "cheating", and an additional big plus. First the trick/cheating.
Downloading the backup isn't the problem. But uploading it if it is needed gets bloody slow with just 5 or 10 Mb/s. So I devised a concept where I split the data into "luke warm" and larger data and "hot" smaller sets. Meaning: "hot" is only what changes frequently (mainly DB) and "luke warm" is what changes rarely (e.g. packages) but tends to be bigger. (For the sake of completeness: "cold" is what is not specific to me and can be easily downloaded elsewhere (like the OS). Both luke warm and hot data gets backed up to an intermediate VPS (at a good provider) with some 200+GB disks (known to be raided) which also serves as a single point (for ease of working) of backups for all my VPSs. From there hot data gets downloaded to my home storage server (the Alix box) every day (in my case a couple of 10 MB) and luke warm data gets downloaded once a week and only the most current set is kept on the VPS.
Additional advantage: always two current backup sets and at different locations/regions.
The additional big plus is that I can long term store some data on blueray, i.e. on an optical medium. In my case that's one single blueray every month. Ultra cheap, really long term and solid and using next to no storage room (physically, i.e. a plastic box with all the BR disks).
Side note: Data of critical importance (mainly company stuff) get written to M-Discs (more expensive than BR but insanely long life and safe).
Oh, and of course that Alix box also serves as local storage/backup for my computers. Seriously important and "holy" stuff like my development work or tax relevant stuff first gets pumped to the SSD and then the Alix writes it out to the Raid set, offering painless speed to me plus Raided backup.
Final remark: All involved boxes, real or virtual, are dual purpose. The Alix box evidently has a main life as router/firewall. And the "intermediate storage server" is used as a normal server (web) just with bigger disk. The backups are timed to be done during the quiet hours. So besides the one time expense for the 2 hard drives (and the enclosure) and maybe 3€/mo for having one good quality VPS with bigger disks there are no extra costs in either money or time.
Oh and: Thank you so much, CloudF#&!% for pestering me about every 10 seconds with your "I'm not a robot" yester-decade security theater BS, making it hard and cumbersome to share knowledge and ideas!
In case someone happens to have a bomb he doesn't need and lives near CF, please kindly drop it at their HQ to rid mankind of that tumor for good.