Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Amazon S3 for server backups
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Amazon S3 for server backups

drdrakedrdrake Member

Hello,

I am using Hetzner's first storage plan (100gb) for some server backups. I am thinking of switching to Amazon S3 because I want to pay for resources I use and not more.

My setup is this:

I backup var-slash-www to a tar archive to mtn-slash-files. I also backups databases using automysqlbackup to mnt-slash-databases. Files are automatically uploaded to my storage plan via webDAV.

I am looking for the best way to backup these files to amazon s3. I am new to amazon s3 and I have read a few articles in this field but I was wondering if any of you have a similar setup and would like to offer suggestions.

«13

Comments

  • PepeSilviaPepeSilvia Member
    edited April 2017

    I use the AWS CLI tool in a shell script that's run via cron. The command is really simple, you just include the name of the file you want to upload, the bucket you want to upload to and it just does it.

    However you should consider Backblaze or OVH instead. They're far cheaper, and one of them is better than the other depending on your usage case. They have similar CLI tools I believe.

    I'd switch, but at my volume the money saved isn't worth the hassle.

    Thanked by 1yomero
  • BharatBBharatB Member, Patron Provider
    edited April 2017

    Install aws-cli in your linux and configure your access key and secret key followed by region, then use the following command

    aws s3 cp filename.tar.gz s3://bucket-name/

    P.S If you don't want to retain a copy locally for saving space you can change cp to mv and that will directly move the file to the bucket.

  • akhfaakhfa Member

    Why don't you use aws glacier? I haven't use it yet but I think it worth to try

  • @akhfa said:
    Why don't you use aws glacier? I haven't use it yet but I think it worth to try

    You would want to kill yourself the day you need to retrieve any data from Glacier. Not only does it take a while, it costs a lot too.

    Thanked by 1yomero
  • @PepeSilvia said:
    I use the AWS CLI tool in a shell script that's run via cron. The command is really simple, you just include the name of the file you want to upload, the bucket you want to upload to and it just does it.

    However you should consider Backblaze or OVH instead. They're far cheaper, and one of them is better than the other depending on your usage case. They have similar CLI tools I believe.

    I'd switch, but at my volume the money saved isn't worth the hassle.
    @BharatB said:
    Install aws-cli in your linux and configure your access key and secret key followed by region, then use the following command

    aws s3 cp filename.tar.gz s3://bucket-name/

    P.S If you don't want to retain a copy locally for saving space you can change cp to mv and that will directly move the file to the bucket.

    I forgot to mention, my scripts deletes all files older than 30 days in the webdav mount. I want to have something similar. I want amazon s3 to work just like this scenario. Is it possible?

    I want to:

    1. Mount the storage as local drive

    2. Compress and add the files directly to amazon s3.

    3. Delete files older than 30 days.

  • BharatBBharatB Member, Patron Provider

    I don't see any reason why you would want it as a local drive though I'm not sure if it's possible, as for deleting files, make sure to give a naming structure to your backups like 2017-04-12.tar.gz etc, use your same script and remove file using
    aws s3 rm s3://bucket-name/filename.tar.gz

  • @BharatB said:
    I don't see any reason why you would want it as a local drive though I'm not sure if it's possible, as for deleting files, make sure to give a naming structure to your backups like 2017-04-12.tar.gz etc, use your same script and remove file using
    aws s3 rm s3://bucket-name/filename.tar.gz

    How could I automate it?

  • BharatBBharatB Member, Patron Provider
    edited April 2017

    @drdrake said:
    How could I automate it?

    write a bash script and cron it weekly

  • @BharatB said:

    @drdrake said:
    How could I automate it?

    write a bash script and cron it weekly

    How can i get which file is older than 30 days if amazon is not mounted?

  • akhfaakhfa Member

    @drdrake said:

    @PepeSilvia said:
    I use the AWS CLI tool in a shell script that's run via cron. The command is really simple, you just include the name of the file you want to upload, the bucket you want to upload to and it just does it.

    However you should consider Backblaze or OVH instead. They're far cheaper, and one of them is better than the other depending on your usage case. They have similar CLI tools I believe.

    I'd switch, but at my volume the money saved isn't worth the hassle.
    @BharatB said:
    Install aws-cli in your linux and configure your access key and secret key followed by region, then use the following command

    aws s3 cp filename.tar.gz s3://bucket-name/

    P.S If you don't want to retain a copy locally for saving space you can change cp to mv and that will directly move the file to the bucket.

    I forgot to mention, my scripts deletes all files older than 30 days in the webdav mount. I want to have something similar. I want amazon s3 to work just like this scenario. Is it possible?

    I want to:

    1. Mount the storage as local drive

    2. Compress and add the files directly to amazon s3.

    3. Delete files older than 30 days.

    For free, you can use s3fs to mount s3. For paid solution, you can use objectivefs. I use objectivefs for production in my website data.

  • BharatBBharatB Member, Patron Provider

    @drdrake said:
    How can i get which file is older than 30 days if amazon is not mounted?

    That's the reason you'll call a list command and itterate through filenames which are actually timestamps lol.

  • I think you can setup Lifecycle Rules on S3 to let S3 automatically removes files older than 30 days (or any number of days you configure it to)

    https://aws.amazon.com/blogs/aws/amazon-s3-object-expiration/

  • Let's wait for scaleway sis ;)

  • ihadpihadp Member

    How much are you really going to save by moving to S3?

  • rclone

  • @ihadp said:
    How much are you really going to save by moving to S3?

    Well, I will use more space and the price will be less than what I am paying. Sounds good to me.

  • I'd recommend Backblaze B2 and their CLI tool. Storage is far, far cheaper

  • Cheaper way is to create a minio server (really simple!), then you'll have a S3-compatible interface at 1/10th of the cost of S3!

    Thanked by 1sergsergiu
  • @spammy said:
    Cheaper way is to create a minio server (really simple!), then you'll have a S3-compatible interface at 1/10th of the cost of S3!

    Where do you host it?

  • @lukehebb said:
    I'd recommend Backblaze B2 and their CLI tool. Storage is far, far cheaper

    What about operations?

  • @drdrake said:

    @lukehebb said:
    I'd recommend Backblaze B2 and their CLI tool. Storage is far, far cheaper

    What about operations?

    I guess there is no such thing on Backblaze. Take a look at here at the bottom: https://www.backblaze.com/b2/cloud-storage-pricing.html

  • @drivex said:

    @drdrake said:

    @lukehebb said:
    I'd recommend Backblaze B2 and their CLI tool. Storage is far, far cheaper

    What about operations?

    I guess there is no such thing on Backblaze. Take a look at here at the bottom: https://www.backblaze.com/b2/cloud-storage-pricing.html

    This is correct - B2 is exceptionally cheap. Its built on the tech that they use for their desktop backups. I've been a customer of theirs for a very long time, and I'm very happy :)

  • moonmartinmoonmartin Member
    edited April 2017

    Why do cost sensitive people still consider Amazon S3 for backup? It's way more expensive than other solutions when you run the numbers for anything other than minimal storage. If your storage requirements are minimal then you can just use google drive for free. Are people bad at math?

    Color me confused.

  • eva2000eva2000 Veteran
    edited April 2017

    moonmartin said: Why do cost sensitive people still consider Amazon S3 for backup? It's way more expensive than other solutions when you run the numbers for anything other than minimal storage. If your storage requirements are minimal then you can just use google drive for free. Are people bad at math?

    Color me confused.

    High availability and reliability and selection of s3 regions closest to your servers. Part of my backup routines has data backups to aws s3 i.e. one aws account has ~1,500+ GB at $23/month with aws s3. AWS IAM management also allows me to assign a unique AWS IAM user per AWS S3 + server pair for security and convenience of managing and revoking AWS IAM user permissions via central AWS Console is great :)

    speed is great too, one of my Centmin Mod users reports 100MB/s backup speeds from OVH BHS to AWS S3 Canada region (ca-central-1) :)

    @ranaldlys said:
    I think you can setup Lifecycle Rules on S3 to let S3 automatically removes files older than 30 days (or any number of days you configure it to)

    https://aws.amazon.com/blogs/aws/amazon-s3-object-expiration/

    yup s3 lifecycle management is what you need, i have it set to 30 days to more from standard to standard-IA storage class then 60 or 90 days to move to Glacier and then XX days for deletion

  • drdrake said: What about operations?

    There's a button at the bottom, "Pricing Organized by API Calls".

  • @eva2000 said:

    moonmartin said: Why do cost sensitive people still consider Amazon S3 for backup? It's way more expensive than other solutions when you run the numbers for anything other than minimal storage. If your storage requirements are minimal then you can just use google drive for free. Are people bad at math?

    Color me confused.

    High availability and reliability and selection of s3 regions closest to your servers. Part of my backup routines has data backups to aws s3 i.e. one aws account has ~1,500+ GB at $23/month with aws s3. AWS IAM management also allows me to assign a unique AWS IAM user per AWS S3 + server pair for security and convenience of managing and revoking AWS IAM user permissions via central AWS Console is great :)

    speed is great too, one of my Centmin Mod users reports 100MB/s backup speeds from OVH BHS to AWS S3 Canada region (ca-central-1) :)

    @ranaldlys said:
    I think you can setup Lifecycle Rules on S3 to let S3 automatically removes files older than 30 days (or any number of days you configure it to)

    https://aws.amazon.com/blogs/aws/amazon-s3-object-expiration/

    yup s3 lifecycle management is what you need, i have it set to 30 days to more from standard to standard-IA storage class then 60 or 90 days to move to Glacier and then XX days for deletion

    All those reasons are things besides cost sensitive. I get why enterprise people who want all that stuff and are willing to pay extra for it would use S3. If cost is the primary concern then S3 is more expensive. Also, google drive and pretty much every other cloud storage service is highly available so no reason to use S3 because of that.

  • moonmartin said: All those reasons are things besides cost sensitive. I get why enterprise people who want all that stuff and are willing to pay extra for it would use S3. If cost is the primary concern then S3 is more expensive. Also, google drive and pretty much every other cloud storage service is highly available so no reason to use S3 because of that

    Haven't really used Google Drive on linux side besides working on Centmin Mod rclone addon https://community.centminmod.com/threads/addons-rclone-sh-client-for-syncing-to-remote-cloud-storage-providers.9299/ so might look into it. Been using AWS S3 for years with awscli, s3cmd and various other linux command line clients for concurrent data transfers etc.

    I also fine AWS S3 speed nice when you have 16+ geographic server locations and need a backup locale close to my server.

  • @drdrake said:

    @spammy said:
    Cheaper way is to create a minio server (really simple!), then you'll have a S3-compatible interface at 1/10th of the cost of S3!

    Where do you host it?

    Any linux machine? https://www.minio.io/

  • @spammy said:

    @drdrake said:

    @spammy said:
    Cheaper way is to create a minio server (really simple!), then you'll have a S3-compatible interface at 1/10th of the cost of S3!

    Where do you host it?

    Any linux machine? https://www.minio.io/

    Amazon S3 offers more for a better price.

  • moonmartinmoonmartin Member
    edited April 2017

    @eva2000 said:

    moonmartin said: All those reasons are things besides cost sensitive. I get why enterprise people who want all that stuff and are willing to pay extra for it would use S3. If cost is the primary concern then S3 is more expensive. Also, google drive and pretty much every other cloud storage service is highly available so no reason to use S3 because of that

    Haven't really used Google Drive on linux side besides working on Centmin Mod rclone addon https://community.centminmod.com/threads/addons-rclone-sh-client-for-syncing-to-remote-cloud-storage-providers.9299/ so might look into it. Been using AWS S3 for years with awscli, s3cmd and various other linux command line clients for concurrent data transfers etc.

    I also fine AWS S3 speed nice when you have 16+ geographic server locations and need a backup locale close to my server.

    All available for a price. There are primarily price sensitive services and then there are services like Amazon s3. Apples and oranges.

    I keep hearing about how S3 is cheap and gotta wonder if I am in some parallel universe or if these people can't do some basic math.

    Thanked by 1quicksilver03
Sign In or Register to comment.