Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In with OpenID
Advertise on LowEndTalk.com

In this Discussion

Amazon Glacier

Amazon Glacier

KuroKuro Member
edited August 2012 in General

http://aws.amazon.com/glacier/

Looks like it could be a much cheaper alternative for those of us who store backups on S3.

Thanked by 3marrco Damian agapitox

Comments

  • But keep in mind: It takes between 3,5 and 4,5 hours to access the data in case of an emergency...

    Got divided by zero. Three times. Feel better ever since...

    Thanked by 3klikli Asim Randy
  • flyfly Member

    it's not meant for emergency backups really.

    backup your stuff to s3 for emergency backups, then sync it to glacier over time.

    Thanked by 1Asim
  • AsimAsim Member

    At this point you may be thinking that this sounds just like Amazon S3, but Amazon Glacier differs from S3 in two crucial ways.

    First, S3 is optimized for rapid retrieval (generally tens to hundreds of milliseconds per request). Glacier is not (we didn't call it Glacier for nothing). With Glacier, your retrieval requests are queued up and honored at a somewhat leisurely pace. Your archive will be available for downloading in 3 to 5 hours.

    Each retrieval request that you make to Glacier is a called a job. You can poll Glacier to see if your data is available, or you can ask it to send a notification to the Amazon SNS topic of your choice when the data is available. You can then access the data via HTTP GET requests, including byte range requests. The data will remain available to you for 24 hours.

    Retrieval requests are priced differently, too. You can retrieve up to 5% of your average monthly storage, pro-rated daily, for free each month. Beyond that, you are charged a retrieval fee starting at $0.01 per Gigabyte (see the pricing page for details). So for data that you’ll need to retrieve in greater volume more frequently, S3 may be a more cost-effective service.

    Secondly, S3 allows you to assign the name of your choice to each object. In order to keep costs as low as possible, Glacier will assign a unique id to each of your archives at upload time.

    http://aws.typepad.com/aws/2012/08/amazon-glacier-offsite-archival-storage-for-one-penny-per-gb-per-month.html

    Thanked by 1djvdorp
  • AsimAsim Member

    All I can think of is store all my mp3, photos etc for emergency-only retrieval (incase all my laptops melt and the home-based backup is blown too)

  • Awesome, completely awesome.

    This will cut our backup costs to S3 by several hundred dollars per month.

    Archive availability in 3 to 5 hours is completely reasonable, as the entire point of backing up customer data is to restore their VPS in the event of a catastrophic server failure. If we have a catastrophic server failure, we're probably going to have more than 3 to 5 hours of work to do before the server is ready for customer data again.

    I am no longer affiliated with IPXcore.
  • Unfortunately, looks like the only high-level APIs available at this time are Java and .NET. But REST is available if you want to roll your own.

    I am no longer affiliated with IPXcore.
  • MrDOSMrDOS Member
    edited August 2012

    Just swung by here to post this and saw it was here already. It's a prime time for Amazon to release something like this; despite the huge number of people using S3 for backups, that's not really what it's for, and the rates for Glacier are more than reasonable. $10/mo per TB of storage is ridiculously good – at current hard drive prices, it would take a year and a half to pay for my own redundant storage of that capacity (2×1TB at $90/drive), and that's not considering the power consumption of whatever those drives are attached to. It's even slightly cheaper per TB than the Backblaze Pod.

    Thanked by 1djvdorp
  • klikliklikli Member
    edited August 2012

    (What's the point of my reply? Please pass)

    Selling multiple 2000-user GApps :) Shoot me a message to make an offer.

  • @fly said: glacier bones you when you try to restore your data. check the restore fees.

    I think Wired wrote up an article on the "restore" fees and worked out it wouldn't be that much overall. The real boning happens when you delete data less than 3 months old: http://aws.amazon.com/glacier/faqs/#How_am_I_charged_for_deleting_data_that_is_less_than_3_months_old

  • Cool, they charge if you delete your data :o Interesting. This whole thing looks like a marketing stunt to put upfront a very small price when in reality it is not that simple. M

    Who's General Failure, and why is he reading my drive A: ?

  • Backup is meant to be write only and not deleted anyway. This pricing makes sure it's cheap only for actual backing up. And in the rare case of emergency a handful of dollars won't kill you.

  • Not really. If I take backups I do them quite often and I would like to delete the ones 3 days back. I think it is meant more as a data bunker to save your files just in case, not a real back-up scenario. M

    Who's General Failure, and why is he reading my drive A: ?

  • considering it's for automated use anyway i see no need to delete stuff so early. but yeah, apparently we agree on the use-case.

  • The 0.03-per-gigabyte-deletion is okay, except I found this today:

    The largest archive that can be uploaded in a single Upload request is 4 gigabytes.

    Which means some of our client data, usually archived per-VPS-container, will need to be split until multiple files.

    @Surge said: @Damian, see http://blog.cloudgates.net/

    I see it, but not sure what i'm supposed to be seeing here...

    I am no longer affiliated with IPXcore.
    Thanked by 1klikli
  • @Maounique said: Not really. If I take backups I do them quite often and I would like to delete the ones 3 days back. I think it is meant more as a data bunker to save your files just in case, not a real back-up scenario.

    Hey M I don't know what we'd use this for either. We just have rolling backups of up to 7 days - not 3 months or 2 years.

    Internap VPS, Web Hosting and more - Cloud Shards | Need a VPS Upgrade?
    Query Foundry, LLC AS62638
  • For example, I was thinking I only have 1 Backup of all my personal movies and pictures and that is some 50 GB. If I upload there will free 50 GB of backup and will cost me 50 cents in the long run. It may be an idea. Otherwise, for a person is not much use. M

    Who's General Failure, and why is he reading my drive A: ?

  • @Maounique said: For example, I was thinking I only have 1 Backup of all my personal movies and pictures and that is some 50 GB. If I upload there will free 50 GB of backup and will cost me 50 cents in the long run.

    it'd be free, not 50 cents. amazon automatically forgives invoices below $1 due to merchant fees

    Postgres

    Thanked by 2Maounique djvdorp
  • @justinb said: it'd be free, not 50 cents. amazon automatically forgives invoices below $1 due to merchant fees

    cool :o So I can use 90 GB in fact for free... Well, now it is really interesting :) Thanks for the tip ! M

    Who's General Failure, and why is he reading my drive A: ?

    Thanked by 1klikli
  • justinbjustinb Member
    edited August 2012

    i actually have no idea if it's still like that, but my past invoices for $0.73 and $0.49 etc are all zeroed out and i was never billed, with valid cards on file and everything

    do tell if you find anything out tho

    image

    Postgres

  • How does someone access that? FTP? RSYNC? Samba? WebUI only?

    Opinions/Posts are to be assumed my own/personal and not company related unless obvious
    Working @ EDIS and owning some others (and/or parts of) | Available for consulting | http://as198412.net | https://william.si

    Thanked by 1djvdorp
  • flyfly Member

    they won't charge you 82 cents because that's almost less than the transaction fee usually

  • NanoG6NanoG6 Member
    edited August 2012

    @justinb said: it'd be free, not 50 cents. amazon automatically forgives invoices below $1 due to merchant fees

    @justinb said: i actually have no idea if it's still like that, but my past invoices for $0.73 and $0.49 etc are all zeroed out and i was never billed, with valid cards on file and everything

    May be still on AWS Free Usage Tier (1 year)? aws.amazon.com/free/

  • No, for real now, how the heck do i get data up there?

    Opinions/Posts are to be assumed my own/personal and not company related unless obvious
    Working @ EDIS and owning some others (and/or parts of) | Available for consulting | http://as198412.net | https://william.si

    Thanked by 1klikli
  • Good question @William

    I use http://tuxlite.com to configure all my VPSes and I love it!

  • @William said: No, for real now, how the heck do i get data up there?

    At the moment, you use either the .NET or the Java SDK.

    I am no longer affiliated with IPXcore.
  • lbftlbft Member

    @Damian said: At the moment, you use either the .NET or the Java SDK.

    Or write your own using the REST API.

  • KuroKuro Member

    In regards to early (before 3 months) deletion, it appears that the cost of storing data for however long plus the cost of early deletion is equal to the cost of simply storing that data for 3 months.

    This makes sense since (from what I can tell) they're using tape storage for this, so the tapes aren't always in a drive, and the tapes only have so many writes before they need replacing. If everyone was deleting everything after 3-7 days (and they weren't charging for early deletion), it would wear out the durability of the tapes much quicker, and the cost of replacement tapes would quickly outweigh the price they charge for the service.

  • MaouniqueMaounique Member
    edited August 2012

    Tapes, OMG... This does explain a few things, however I would NOT trust my data to ONE tape. I hope they do have "RAIT" of some sort :P M

    Who's General Failure, and why is he reading my drive A: ?

  • Maounique,

    Tapes these days are generally large SATA/SAS disks. Think of it like eSATA on a server(in some cases it is).

    Unfortunately if that is the case, the it is a single disk being shared and has the possibility of the data being lost, even under the most perfect of conditions =(.

    ChicagoVPS.net - OpenVZ/Xen Based - SolusVM Control Panel - TUN/PPP/FUSE/SIT/GRE - cPanel/DirectAdmin/Parallels - Chicago/Buffalo/LA Coming Soon! - Great Support!

  • @Maounique

    Amazon Glacier is designed to provide average annual durability of 99.999999999% for an archive. The service redundantly stores data in multiple facilities and on multiple devices within each facility. To increase durability, Amazon Glacier synchronously stores your data across multiple facilities before returning SUCCESS on uploading archives.

    SimpleNode | Minecraft and VPS Hosting | KVM and OpenVZ | Dallas, TX and Phoenix, AZ | PayPal and BitCoins accepted

  • Aha, thanks :) I will think of it, ATM I have the backup in another house in another town, might be worth it 50 cents a month or so for peace of mind. Or even free :) M

    Who's General Failure, and why is he reading my drive A: ?

  • dnomdnom Member

    Nowhere it says on their site that they are using tapes though. Also:

    Amazon Glacier is designed to provide average annual durability of 99.999999999% for an archive. The service redundantly stores data in multiple facilities and on multiple devices within each facility. To increase durability, Amazon Glacier synchronously stores your data across multiple facilities before returning SUCCESS on uploading archives. Unlike traditional systems which can require laborious data verification and manual repair, Glacier performs regular, systematic data integrity checks and is built to be automatically self-healing.

    I think they are using HDDs. I think they're setup is that they power off the disk(probably the file server) after writing to save on energy costs. That might explain the charge for early deletion and the waiting period before getting your data.

  • anyone wants to write an app to simply upload files from bash? :p

    Opinions/Posts are to be assumed my own/personal and not company related unless obvious
    Working @ EDIS and owning some others (and/or parts of) | Available for consulting | http://as198412.net | https://william.si

    Thanked by 1djvdorp
  • Internap VPS, Web Hosting and more - Cloud Shards | Need a VPS Upgrade?
    Query Foundry, LLC AS62638
    Thanked by 1dnom
  • @Kuro said: In regards to early (before 3 months) deletion, it appears that the cost of storing data for however long plus the cost of early deletion is equal to the cost of simply storing that data for 3 months.

    Yep, and if you keep your data for less than a month, the price becomes 4 cents per gigabyte (1 cent for storage, 3 cents for deletion). Still cheaper than S3.

    I am no longer affiliated with IPXcore.
  • KuroKuro Member

    @Damian said: Yep, and if you keep your data for less than a month, the price becomes 4 cents per gigabyte (1 cent for storage, 3 cents for deletion).

    Where did you read that at? AFAIK both the storage and deletion fees are pro-rated.

  • When you make a retrieval request, a robotic arm grabs the tape with your data in, slots the tape into a drive, and then your data will be transferred to a hard drive ready for you to access. The 3-5 hour window is simply be the time it takes for the robotic arm to become available — if a lot of people are making retrieval requests, you have to wait in line.

    That sounds awesome... lol.

    SimpleNode | Minecraft and VPS Hosting | KVM and OpenVZ | Dallas, TX and Phoenix, AZ | PayPal and BitCoins accepted

  • DamianDamian Member
    edited August 2012

    @Kuro said: Where did you read that at?

    " In the US East (Northern Virginia) Region, you would be charged a prorated early deletion fee of $0.03 per gigabyte deleted within three months. "

    From: http://aws.amazon.com/glacier/faqs/#How_am_I_charged_for_deleting_data_that_is_less_than_3_months_old

    It then goes down per month. Between 1 and 2 months, 2 cents. Between 2 and 3 months, 1 cent. After 3 months, I think it's free.

    Pricing for storage ends up being ~$0.0003 per gigabyte per day (as defined in http://aws.amazon.com/glacier/faqs/#How_is_my_storage_charge_calculated ), so storing a 4 gigabyte archive for 7 days would end up being:

    ($0.0003 x 7) x 4) =$0.0084 (storage fee) + $0.03 (early deletion fee) equals $0.0384 total

    Rounded up to 4 cents. Not bad, and still cheaper than S3 (even considering S3 RR), if you don't need S3's instant availability.

    I am no longer affiliated with IPXcore.
  • KuroKuro Member

    @Damian said: " In the US East (Northern Virginia) Region, you would be charged a prorated early deletion fee of $0.03 per gigabyte deleted within three months. "

    Maybe they do charge you a full cent per month, but going by the way they pro-rate everything else...

    Either way, still much cheaper than S3 for long term storage.

  • @William said: anyone wants to write an app to simply upload files from bash? :p

    I am thinking about doing this. You might contact me by PM for wishes/needs/requirements in your opinion. It will be my spare time project without a fixed deadline, but Glacier looks useful for me too :)

    I use http://tuxlite.com to configure all my VPSes and I love it!

  • Thought about setting up a full array of backup systems and offering backup space at $10 - $15 /TB monthly with FTP, RSYNC and a web based file manager... It'd be something to the effect of 4 TB drives in raid1 x 12 sets per system. This would cost us roughly $8.50 per TB plus support per month if we assumed that the systems would only last 1 year before exploding...

    BlueVM | Best VPS Deals [~] 1GBPS, RAID-10, OpenVZ/KVM, 8 amazing locations. [~]
  • @BlueVM said: This would cost us roughly $8.50 per TB plus support per month if we assumed that the systems would only last 1 year before exploding...

    Would definitely be interested in LEB version of this considering it'd be available at multiple locations for redundancy. We currently just use LEB backup servers.

    Internap VPS, Web Hosting and more - Cloud Shards | Need a VPS Upgrade?
    Query Foundry, LLC AS62638
  • @Surge said: see http://blog.cloudgates.net/

    https://groups.google.com/d/topic/cloudgates/Co9zWvCSqHU/discussion

    Initial support for Glacier is now deployed.

    Only uploads work at this point.

Sign In or Register to comment.