Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


VPS Needed for Large Data Transfer Job (~1.5TB/day for 2-3 months)
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

VPS Needed for Large Data Transfer Job (~1.5TB/day for 2-3 months)

MasonRMasonR Community Contributor

Looking for a VPS to complete a data transfer sync from one Google Drive account to another (approx. 60TB in total) using rclone. Specs don't really matter, but the VPS will use a consistent 150 Mbit/s (hard limit) for about 2-3 months until the job is complete.

VZ Type: Doesn't matter
Number of Cores: Doesn't matter
RAM: >=512MB
Disk Space: Doesn't matter
Disk Type: Doesn't matter
Bandwidth: Unmetered or ridiculously high, like 60TB
Port Speed: >=200 Mbps
DDoS Protection: Doesn't matter
Number of IPs: 1 IPv4
Location: Doesn't matter
Budget: Hoping to keep this <=$5/mo, but flexible
Billing period: Monthly

Please post offers and recommendations. Please don't post if you aren't sure if network "fair use" policies will permit this amount of usage. Data isn't being stored locally, so disk size doesn't matter. The data is being encrypted upon upload, but even one CPU on an atom machine would be plenty. Port speed and bandwidth are really the only things that matter for this. Would like to have the VM spun up and rolling by EOD tomorrow (23 Aug). Thanks!

«13

Comments

  • HarambeHarambe Member, Host Rep

    If my /r/datahoarder memory serves me right, I think people generally used scaleway for this type of transfer. Or Google Cloud because of the free internal bandwidth.

    Thanked by 4MasonR Ksygo dosai 479555
  • MasonRMasonR Community Contributor

    @Harambe said:
    If my /r/datahoarder memory serves me right, I think people generally used scaleway for this type of transfer. Or Google Cloud because of the free internal bandwidth.

    Scaleway might do the job nicely if they don't send passive aggressive emails to cut back on network usage (I know online.net "unlimited" servers are subject to a fair use policy). The baremetal C1 ARM box would probably do the job at €2.99/mo. I don't particuarly like online.net after the whole price raise a couple months after the promo they did a while back, so this will be my fallback should nothing else work out. Thanks for the recommendation!

    Google Cloud scares me a bit due to their complex pricing and crazy bandwidth overage bills. Indeed, all traffic should be internal to Google, but for peace of mind I'd rather stay away and fork out the $5/mo instead.

    Thanks again! :)

  • bugrakocbugrakoc Member
    edited August 2019

    I have used Tilaa.com for several years and they are solid. They are what comes to my mind when unlimited traffic is mentioned :smile: I'll paste their cheapest specs below.

    Server € 6.00 / mo
    Memory 1 GB
    Storage 20 GB HDD
    Swap 1 GB
    CPU cores 1 × 2.4Ghz @ 80%, included
    Multiple snapshots yes / first is free
    Network traffic unlimited / free

    Edit: It's KVM.

    Thanked by 1MasonR
  • TheLinuxBugTheLinuxBug Member
    edited August 2019

    bugrakoc said: I have used Tilaa.com for several years and they are solid. They are what comes to my mind when unlimited traffic is mentioned I'll paste their cheapest specs below.

    Be sure to note that it is NOT unlimited traffic but 'fair-share' and they spell this out very specifically in their AUP / TOS. Their definition of 'fair-share' is 5TB, if you exceed that and become abusive they will write to you and request you stop the traffic, upgrade, or agree to be limited. I have only ever got around the limit once and been notified, generally if your realistic and not abusive they won't bother you and their routing is quite good.

    That said, I wouldn't use the above outlined server for what you are wanting as it sounds like you would quickly exceed that 'fair-share' limit.

    my 2 cents.

    Cheers!

    Thanked by 1MasonR
  • Maybe check with @exception0x876 which of Wishosting's unmetered 250/500M nodes have lowest average utilization, and order that plan

    As I understand you won't be thrashing disks; just using RAM, 150mbit sustained (75+75?) and a fraction of the available cpu crypto capacity (decrypt+encrypt combined).

    My storage VM in CA always seems to have ~500mbit available .

    Thanked by 1MasonR
  • williewillie Member
    edited August 2019

    Two OVH VPS at 100 mbit each. It's crazy if there's no way to do that transfer completely inside google though. Or that there's scary billing for Google Cloud but not Google Drive.

    Thanked by 1MasonR
  • FHRFHR Member, Host Rep

    I can do it in CZ for 4.5€/month (~$5). I have Google on IX so traffic to it is basically free (not to mention it's a direct ~0.7ms route), for your purpose I can hard limit you to 350Mbps and feel free to max it out 24/7.

  • MasonR said: Please don't post if you aren't sure if network "fair use" policies will permit this amount of usage.

    Sorry about doing the only thing you didn't want to be done :tongue:

  • sanvitsanvit Member
    edited August 2019

    I'm hitting around 1.4TB/day (750GB IN+OUT) with BuyVM $2/m slice and haven't had any issues

    Thanked by 1MasonR
  • YmpkerYmpker Member
    edited August 2019

    @MasonR If it's mostly .iso files why not do the native file copying implemented in Google Drive? On your main gdrive account share the folder(s) with all the files with your backup gdrive account.
    Once it's shared, login with your backup gdrive account and add the shared folder to quick access. Now, ctrl+a select all files/isos in each folder, right click and select "make copy". Now it will make a copy of all the selected files in your local (backup) gdrive account. Once done, you can disable the shared folder again on your main account. Basically this https://zapier.com/learn/how-to/copy-google-drive-folders/ just with a shared folder.

    I know this takes some manual work, but it would keep bandwidth internal. Just an idea. Perhaps I'm a fool as I've never done it to that extent but moving the files over a couple months via vps seems also quite the hassle.

    Thanked by 1MasonR
  • NeoonNeoon Community Contributor, Veteran

    Please no, its fucking clickbait, thats all it is.

  • @Neoon said:

    Please no, its fucking clickbait, thats all it is.

    Good to know.

  • NeoonNeoon Community Contributor, Veteran

    Just pick the french, they are fine if you use 200Mbit for porn transfer all the day.

  • NekkiNekki Veteran

    @Neoon said:
    Just pick the french, they are fine if you use 200Mbit for porn transfer all the day.

    I can vouch for this.

    Thanked by 2bugrakoc MasonR
  • If you really don't want to pay offline.net, you may consider get a slice from buyvm. I recommend to check with @Francisco before, but a 2GB should be ok with 200 Mbps all days.

    Thanked by 1MasonR
  • I would suggest BuyVM as well. As far as I know the slices have unmetered bandwidth.

    Thanked by 1MasonR
  • HarambeHarambe Member, Host Rep

    I would've suggested BuyVM as well, but afaik you're supposed to keep your 24/7 utilization around 100Mbps per 4GB of ram - and a 4GB slice is 3x his budget. Would need to ask Fran for an exception to do it on a smaller plan I think.

    Thanked by 2MasonR Ympker
  • Google Cloud.
    You may even already have free credits. Block all external traffic via rules and copy away; should be super-fast too.

    Otherwise OVH and subsidiaries, they really don't seem to care if you hit your limit 24/7. Just don't expect burst speed to be available afterwards (which doesn't seem to matter anyway)

    At one point you could even server-side copy large amounts of data - but that now seems to have a limit, even lower than the 750GB/day one. That's still per "user", however.

    With Google Cloud you'll be able to do 60TB in much less time with certain configurations, if time is important.

    Thanked by 1MasonR
  • I've synced around 20 TB using rclone on the free tier with no charges. It will spike to 50-80MB/s for the first minute or so before sustaining around 20MB/s. This is on us-east.

    Thanked by 3MasonR SirFoxy Ouji
  • MasonRMasonR Community Contributor

    @willie - considered that, but would rather keep things simple on a single machine. Fire and forget is the goal.

    @bugrakoc - heh, no worries :)

    @sanvit + @beagle - as @Harambe mentioned, I believe there's some implied limits (not sure what they are, though). Maybe Fran can shed some light. I do already have a $2/mo slice for the Blesta goodness, though.

    @Ksygo - time isn't important, so I'd rather not do the team drive/service account trickery. Or anything manual for that matter. A simple sync is perfect as I don't really care about how long it takes and I can limit it just under 750GB/day so I don't get the account temp banned.

    Would rather do it outside of google cloud for peace of mind. If I need to hire a lawyer and two financial advisors to decipher their pricing, I'd rather just stay away even if it'll end up being free.

    @CyberneticTitan - it's gonna take the same amount of time whether I do it at full speed and cut it off or just do a slow sync due to the 750GB/day upload limit per account.

    Thanked by 2beagle Ksygo
  • williewillie Member
    edited August 2019

    MasonR said: limit it just under 750GB/day so I don't get the account temp banned.

    This sounds like 100mbps is enough. It's about 1TB/day. Oh hmm, don't know if it's full duplex: I guess I could try on one of my kimsufi dedis.

  • Do what @Kysgo recommended, I done the exact same thing. It transfers at 1G down and 1G up and bandwidth isn't counted within the Google universe so it's free of charge if you have free credits.

  • hetzner cloud instance 30TB month ? With the cap at 750GB/day, it shoul be ok.

    Thanked by 2MasonR Hetzner_OL
  • MasonRMasonR Community Contributor
    edited August 2019

    @willie said:

    MasonR said: limit it just under 750GB/day so I don't get the account temp banned.

    This sounds like 100mbps is enough. It's about 1TB/day.

    It's transferring from one account to another, so essentially 750GB download + 750GB upload daily

    @tester4 said:
    Do what @Kysgo recommended, I done the exact same thing. It transfers at 1G down and 1G up and bandwidth isn't counted within the Google universe so it's free of charge if you have free credits.

    Needs to be limited to not breach the daily cap. Speed isn't the issue as long as it allows 8.5MB/s down + 8.5MB/s up sustained.

  • williewillie Member
    edited August 2019

    Hmm as someone else says, Hetzner Cloud maybe? I think it's limited to 20TB/m outbound, but inbound doesn't count. Port speed is now 1gbit I think. Was 10gbit in the past.

    Thanked by 2MasonR Hetzner_OL
  • @MasonR said:
    @sanvit + @beagle - as @Harambe mentioned, I believe there's some implied limits (not sure what they are, though). Maybe Fran can shed some light. I do already have a $2/mo slice for the Blesta goodness, though.

    @Francisco did say he's OK with pushing 150TB on some other thread (not sure of the exact pricing of the slice though). Maybe a little hint please fran?

  • We'd be glad to sponsor this if any LET provider wants to chime in and make an offer. @MasonR is a giver, not a taker. Please, someone get him operational within his deadline of tomorrow and we'll foot the bill. Premium guys welcome. No budget cap as long as it's reasonable.

    His call on who he chooses.

  • @Clouvider @Francisco @intovps @HostDoc tagging you because of his short deadline. We'll take care of it if you can help him out. Sorry if I missed others.

    Thanked by 3sanvit MikePT intovps
  • FranciscoFrancisco Top Host, Host Rep, Veteran

    @sanvit said:

    @MasonR said:
    @sanvit + @beagle - as @Harambe mentioned, I believe there's some implied limits (not sure what they are, though). Maybe Fran can shed some light. I do already have a $2/mo slice for the Blesta goodness, though.

    @Francisco did say he's OK with pushing 150TB on some other thread (not sure of the exact pricing of the slice though). Maybe a little hint please fran?

    @masonR - I can put you on a very quiet node if you want to pickup a 512MB or similar with a slab.

    Francisco

Sign In or Register to comment.