Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Amazon S3 for server backups - Page 2
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Amazon S3 for server backups

2

Comments

  • moonmartin said: I keep hearing about how S3 is cheap and gotta wonder if I am in some parallel universe or if these people don't understand basic math.

    Cheap is always relative to the person's perspective and their relative alternatives based on their needs i.e. easy of deployment, variety of command line, mounting options and high availability/reliability etc.

    AWS S3 Standard-IA storage class is $0.0125/GB so $12.50/TB per month.

  • moonmartinmoonmartin Member
    edited April 2017

    @eva2000 said:

    moonmartin said: I keep hearing about how S3 is cheap and gotta wonder if I am in some parallel universe or if these people don't understand basic math.

    Cheap is always relative to the person's perspective and their relative alternatives based on their needs i.e. easy of deployment, variety of command line, mounting options and high availability/reliability etc.

    AWS S3 Standard-IA storage class is $0.0125/GB so $12.50/TB per month.

    You 'forgot ' to include data transfer. Sure it was just coincidence. Also the fine print.

    Standard - Infrequent Access Storage has a minimum billable object size of 128KB. Smaller objects may be stored but will be charged for 128KB of storage. Standard – Infrequent Access Storage is charged for a minimum storage duration of 30 days. Objects that are deleted, overwritten, or transitioned to a different storage class before 30 days will incur the normal usage charge plus a pro-rated request charge for the remainder of the 30 day minimum. Objects stored 30 days or longer will not incur a 30-day minimum request charge.

    A more honest argument should probably use standard storage price and also include data transfer.

    Using amazon cost calculator with 1TB transfer in per month and 1TB inter region transfer out is $225

    1TB transfer in and 1TB transfer out using cloudfront it's $64 per month.

    If you don't transfer that much then of course it is much less but still way more than the cost you are implying by at least double...at a minimum.

  • eva2000eva2000 Veteran
    edited April 2017

    Yeah i know.. use AWS Life Cycle management to manage AWS S3 storage class migrations automatically between Standard > Standard IA and Glacier with 1500GB at ~US$23/month. Also AWS S3 is pay as you go so at some data set sizes, alternatives like Google Drive might be more expensive i.e. if you data set was 2100GB. As far as I can see for my Google Drive, I only have plan options for 1TB, 2TB, 10TB. So for 2100GB, I'd be paying for a 10TB account! Hence, why the term 'cheap' is relative to one's own alternatives :)

    Thanked by 1Dormeo
  • S3 supports automatic retention in the console or cli. Its literally a few clicks to setup your retention policy.

    Check the docs.

  • WSSWSS Member

    Fuck Bezos.

  • moonmartinmoonmartin Member
    edited April 2017

    @eva2000 said:
    Yeah i know.. use AWS Life Cycle management to manage AWS S3 storage class migrations automatically between Standard > Standard IA and Glacier with 1500GB at ~US$23/month. Also AWS S3 is pay as you go so at some data set sizes, alternatives like Google Drive might be more expensive i.e. if you data set was 2100GB. As far as I can see for my Google Drive, I only have plan options for 1TB, 2TB, 10TB. So for 2100GB, I'd be paying for a 10TB account! Hence, why the term 'cheap' is relative to one's own alternatives :)

    Or just don't use Amazon and their pay for every little thing and do everything our way service that ends up costing much more than you think if you just use a little math.

    Up to you. Now you are saying "relatively". So I guess the goal posts moved. It's not cheap relative to anything. It's one of the more expensive cloud services out there. What it IS good at is making you think it's cheap (relative, absolute, in comparison to, take your pick) assuming you actually believed that $0.0125/GB price you used.

  • Saving money is the last reason to do anything on S3. It will definitely cost more than LET storage for anything of any size. That said, Hetzner's smaller storage plans are overpriced: they get better at 2TB and above.

    For a small amount like 100GB, I'd just get a storage VPS. How much storage do you actually want to use? wishosting.com has a 40GB NAT storage vps for something like 3 euro per YEAR. Maybe you could get a couple of those. Or @pbgben mentioned having KVMs at OVH Gravelines at $1 per 100GB + fees (whatever those are). Serverhub.com, 125GB for $15/year at many locations.

    Or if you want archive storage, I like C14 Intensive which is 0.005 euro/GB/month with free upload/download/delete. The non-intensive alternative is 0.002 euro/GB/month for storage, but upload/download/delete cost 0.01/GB each, so it only wins if you leave the stuff there without touching it for a long time. OVH Cloud Archive is similar except there's no deletion fee, which is good if you want to delete your backups after some specified retention period.

    Again the amounts involved are so piddly that it doesn't seem worth worrying about. This is more interesting if you have many TB to deal with.

  • @WSS said:
    Fuck Bezos.

    Stay classy, WSS...

    Valuable contributions such as this, keeps me more and more away from this forum.

  • WSSWSS Member
    edited April 2017

    @Weblogics said:

    @WSS said:
    Fuck Bezos.

    Stay classy, WSS...

    Valuable contributions such as this, keeps me more and more away from this forum.

    Bye, Jeff. Not sure if we can find someone else to post 8 things from their RSS feed in several years, but we'll manage.

  • reetwoodreetwood Member
    edited April 2017

    @WSS said:
    Fuck Bezos.

    u r vry smart....

    I've found backblaze (b2) to be awfully slow for upload. I wouldn't recommend it, unless you are really cheap.

    I'll have to give OVH's block storage a play with.

    s3 costs are peanuts considering what you get :)

  • eva2000eva2000 Veteran
    edited April 2017

    moonmartin said: assuming you actually believed that $0.0125/GB price you used.

    not believed, it's real - it's right in my AWS S3 billing reports

    Cheap will always be 'relative' term for each person. My criteria is at least 50-100MB/s transfer speeds for backups for 16+ geographic locations of my servers and pay as you go billing with secure per server user control/revoke access from central console.

    That criteria rules out slow Kim Sufi capped at 100Mbps and OVH capped at 250-500Mbps with at least $105/month extra to go to 1Gbps https://www.ovh.com/us/dedicated-servers/bandwidth-upgrade.xml and any other dedicated or vps host below <1Gbps already. Cheap IS relative to ones needs!

  • eva2000eva2000 Veteran
    edited April 2017

    moonmartin said: Using amazon cost calculator with 1TB transfer in per month and 1TB inter region transfer out is $225

    1TB transfer in and 1TB transfer out using cloudfront it's $64 per month.

    If you don't transfer that much then of course it is much less but still way more than the cost you are implying by at least double...at a minimum.

    AWS S3 out to internet is US$90/TB for first 10TB for US destinations https://aws.amazon.com/s3/pricing/ and as i said 'cheap' is relative as I don't transfer out much - it's storage and transfer speeds that matters and the variety of ssh command line clients and integrations available for the storage layer for my specific needs.

    also look at per region egress traffic examples

    1TB egress out

    to US region

    • AWS S3 = US$90/TB for 1st 10TB
    • Google Cloud = US$120/TB for 1st 10TB

    to Sydney region

    • AWS S3 = US$140/TB for 1st 10TB
    • Google Cloud = US$190/TB for 1st 10TB

    to Singapore region

    • AWS S3 = US$120/TB for 1st 10TB
    • Google Cloud = US$120/TB for 1st 10TB

    to Hong Kong region

    • AWS S3 = US$120/TB for 1st 10TB connecting to Singapore
    • Google Cloud = US$230/TB for 1st 10TB

    So 1.5TB data egress out to US + Sydney + Singapore + Hong Kong region would cost

    • AWS S3 = 1.5x (90+140+120+120) = US$705
    • Google Cloud = 1.5x (120+190+120+230) = US$990

    Again 'cheap' is relative :)

  • @drdrake: AWS S3 have versioning and lifecycle rules, which I think much faster, consuming less resource (and also cheaper) than uploading full compressed backups in daily basis. I've been using rclone (https://rclone.org/s3/) to do daily sync from the server to S3, and enabled the versioning and lifecycle rules in S3 to keep and remove older version of the files. For point in time recovery/restore from the backup files, you can use this: https://github.com/madisoft/s3-pit-restore. For me, this is an incremental backup on steroid ;).

  • @godong said:
    @drdrake: AWS S3 have versioning and lifecycle rules, which I think much faster, consuming less resource (and also cheaper) than uploading full compressed backups in daily basis. I've been using rclone (https://rclone.org/s3/) to do daily sync from the server to S3, and enabled the versioning and lifecycle rules in S3 to keep and remove older version of the files. For point in time recovery/restore from the backup files, you can use this: https://github.com/madisoft/s3-pit-restore. For me, this is an incremental backup on steroid ;).

    I can not seem to find the option to remove files older than 30 days. Any help?

  • drdrake said: I can not seem to find the option to remove files older than 30 days. Any help?

    http://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html

  • xyzxyz Member
    edited April 2017

    eva2000 said: Cheap IS relative to ones needs!

    No, you're just re-defining 'cheap' to suit what you want it to in your reality distortion field.

    Yes, if you cherry pick features that only Amazon can provide, or have extremely niche requirements (unlikely), or just are aboard the Amazon hype train (likely), then Amazon will be the best solution for you. But for 99.99% of others, Amazon isn't cheap by any stretch of the (sane) imagination.

    I get that people like to justify extravagant purchases by convincing themselves that they "need" 500+Mbps bandwidth for backups, or "need" a 4 server cluster run a simple Wordpress blog with 10 visits a day, or whatever gives them a hard, and then pride themselves on how cheaply they can meet their requirements. But I think that the LET mindset is generally able think more minimalistically and separate the needs from the nice-to-haves. By which token, Amazon is not cheap.

    Thanked by 3Lee WSS southy
  • +1 times a million for Backblaze B2. Last October I sat down with them to discuss managing their new Phoenix datacenter and got a lot of insight on their operation. B2 is cheaper to boot, too.

  • JTRJTR Member

    @reetwood said:
    I've found backblaze (b2) to be awfully slow for upload. I wouldn't recommend it, unless you are really cheap.

    At work we recently started using it to back up the office server and it's quite speedy for the upload - we have a shared gigabit fiber connection and B2 was able to burst to full line speed at times although it typically fluctuated between a fairly wide range (I'd say typical upload speed was 300-800mbps?). It wasn't even very well optimized for performance either in the settings, if connection threads had been set higher it would have been faster (we did run it at two different thread count settings and observed a significant upload performance improvement with the higher settings).

    At home I use Backblaze on my personal computer and it easily saturates my line's upload. I'm quite happy with it.

    When we were discussing backup options prior to choosing B2 our #1 alternative pick was Hetzner. I don't think we had a firm #2 alternative option though... Our requirements ruled out a number of other candidates though that would have been fine if it was for personal use or a smaller amount of data.

  • xyz said: Yes, if you cherry pick features that only Amazon can provide, or have extremely niche requirements (unlikely), or just are aboard the Amazon hype train (likely), then Amazon will be the best solution for you. But for 99.99% of others, Amazon isn't cheap by any stretch of the (sane) imagination.

    You're still defining 'cheap' relatively even by the 'LET mindset' as that mindset is just another set of lower minimalist 'criterias' as opposed to my specific criterias. If you don't need all the high end features that AWS offers, then yes alternatives relative to your needs are cheaper.

  • eva2000eva2000 Veteran
    edited April 2017

    Damian said: +1 times a million for Backblaze B2. Last October I sat down with them to discuss managing their new Phoenix datacenter and got a lot of insight on their operation. B2 is cheaper to boot, too.

    Do they still only have one datacenter in USA ? The Last time I evaluated B2 it was very slow for Asian and Australia transfers.

  • @drdrake said:

    @spammy said:

    @drdrake said:

    @spammy said:
    Cheaper way is to create a minio server (really simple!), then you'll have a S3-compatible interface at 1/10th of the cost of S3!

    Where do you host it?

    Any linux machine? https://www.minio.io/

    Amazon S3 offers more for a better price.

    Are you sure? In LET you can get 1TB of storage for 7 bucks a month....

  • I guess I still don't understand what's being sought here. If the idea is to decrease expenditures for a small amount of storage, S3 is exactly the wrong way to do it, because of both the storage and bandwidth charges, even if you pay for less actual storage.

  • eva2000eva2000 Veteran
    edited April 2017

    @willie said:
    I guess I still don't understand what's being sought here. If the idea is to decrease expenditures for a small amount of storage, S3 is exactly the wrong way to do it, because of both the storage and bandwidth charges, even if you pay for less actual storage.

    again it's relative ^_^ :)

    If the user was in say India, Hong Kong or Singapore with 100GB of backup data - would you be recommending solely USA based backup providers or providers who have multi-region support closer to the end user ?

    Does no one care about speed of their backups (and locale of their backup servers) ? And thus ultimately how fast you restore your data in case of emergency ?

    My criteria is to be able to reload OS and restore my site and data config within 15-30 minutes ideally for small-medium sites.

  • @spammy, thanks for minio. Is Awesome!

  • eva2000 said: If the user was in say India, Hong Kong or Singapore with 100GB of backup data - would you be recommending solely USA based backup providers or providers who have multi-region support closer to the end user ?

    Well, OP was unsatisfied with Hetzner due to paying 2.90 euro/month for 100GB of storage while not using the whole 100GB. So we have a cost-sensitive user who is ok with an EU backup location. S3 doesn't sound like the right thing. There are in fact some LET hosts with plans at the locations you mention.

    As for restore speed, I don't think it should be that important. Some of my own backup stuff is in C14 which takes hours to retrieve. More is on a Kimsufi KS-2E which has 2TB of disk on a 100mbps port, so 2 full days to download. If you have a production service that can't stand a longish interruption, you want a hot spare server rather than a pile of data in remote storage.

    Thanked by 1Lm85H4gFkh3wk3
  • willie said: Well, OP was unsatisfied with Hetzner due to paying 2.90 euro/month for 100GB of storage while not using the whole 100GB. So we have a cost-sensitive user who is ok with an EU backup location. S3 doesn't sound like the right thing. There are in fact some LET hosts with plans at the locations you mention.

    I see really depends on what original post user wants or finds an acceptable compromise i.e. speed of transfers etc. But 100GB is only US$2.30/month on AWS S3 Standard storage class or US$1.25/month on AWS S3 Standard-IA storage class. Though egress out of AWS S3 would be ~$9 for 100GB data.

    I'm not saying AWS S3 is only option but it is the option the original poster is looking to evaluate. Just hope he understands the value of reliable backups as you only realise that value when you most need it i.e. data recovery. I also use LET hosts for dedicated data backup servers in conjunction with AWS S3 so as to not put all my eggs in one basket.

    willie said: As for restore speed, I don't think it should be that important. Some of my own backup stuff is in C14 which takes hours to retrieve. More is on a Kimsufi KS-2E which has 2TB of disk on a 100mbps port, so 2 full days to download. If you have a production service that can't stand a longish interruption, you want a hot spare server rather than a pile of data in remote storage.

    Indeed.. though that would be out of original posters budget to have hot spares so he's fully relying on the speed of backup server's transfers for restores.

  • tarsnap.com is supposed to be the most hardcore backup service, fwiw.

  • WSSWSS Member

    @willie said:
    tarsnap.com is supposed to be the most hardcore backup service, fwiw.

    Tarsnap runs on UNIX-like operating systems (BSD, Linux, MacOS X, Cygwin, etc).

    That alone puts me on edge, and I haven't bothered to look at their code.

  • WSS said:

    That alone puts me on edge, and I haven't bothered to look at their code.

    The guy who does Tarsnap (and wrote all the code) is smarter than hell so I'm sure he got all that stuff right.

  • WSSWSS Member

    @willie said:
    The guy who does Tarsnap (and wrote all the code) is smarter than hell so I'm sure he got all that stuff right.

    Cygwin is bad mojo. One of the things it really fails on are pipes. That tends to make me not trust it for any backup needs. Brilliant programmers can still be bad at other things. :)

Sign In or Register to comment.