Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


VPS Needed for Large Data Transfer Job (~1.5TB/day for 2-3 months) - Page 2
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

VPS Needed for Large Data Transfer Job (~1.5TB/day for 2-3 months)

2

Comments

  • MikeAMikeA Member, Patron Provider
    edited August 2019

    @DA_Mark said:
    @Clouvider @Francisco @intovps @HostDoc tagging you because of his short deadline. We'll take care of it if you can help him out. Sorry if I missed others.

    I'll gladly take a free DA license instead :wink:

    Edit - I'm not serious, don't worry, DirectAdmin is cheap and I respect dev costs. He has an offer from me and many others, I'm sure at the very least the Google Cloud thing would work for him.

    Thanked by 2DA_Mark MasonR
  • MasonRMasonR Community Contributor

    Thanks all! And thank you for the kind gesture, @DA_Mark! Going to review everything in the morning when I wake up and set up everything from there :)

    Thanked by 1Ympker
  • if you can live/work with a natted VM I can set you up for free as I have quite some ressources spare on a servdiscount dedi (aka myloc, unmetered).
    port forwardings if needed shouldn't be a problem...
    can also set that speed limit for network on the node for you, so you don't need to worry.

    other than that I totally agree, hetzner cloud most likely is the cheapest option.

    just let me know ;-)

  • NeoonNeoon Community Contributor, Veteran

    Well, we had MyLoc as a pure backup machine, since after a bit, the backups took way to long. Turns out, they throttled the traffic since it goes via Frankfurt.

    After a few days long ticket, they took care and removed the throttle.
    Even if its advertised as 1000Mbit Unlimited, which is again pure clickbait, like the the russian guy selling his VPS for 1.4$.

    But maybe they stopped doing it, no idea.
    I mean we had maybe 50GB per day or so, not much.

    Thanked by 2Falzo MasonR
  • FalzoFalzo Member
    edited August 2019

    @Neoon said:
    Well, we had MyLoc as a pure backup machine, since after a bit, the backups took way to long. Turns out, they throttled the traffic since it goes via Frankfurt.

    After a few days long ticket, they took care and removed the throttle.
    Even if its advertised as 1000Mbit Unlimited, which is again pure clickbait, like the the russian guy selling his VPS for 1.4$.

    But maybe they stopped doing it, no idea.
    I mean we had maybe 50GB per day or so, not much.

    thanks for the headsup!

    servdiscount has quite a big banner directly on the main page promising 300mbit guarantued and no throttling. though, I never get close to use this for more than a few hours every other month with my normal traffic... I also use that box only for storage, the rest is idling along.

    so who knows, myloc hasn't had the best reputation for years, maybe it really is just marketing and they start throttling anyway and blame whatever instead.
    on the other hand, if the usage is already going to be restricted to ~100ish mbit full duplex - how low do they want to go when having to justify it somehow ^^

    however I also agree that it might be more of a 'guarantue' after all, if you pick a provider with a clearly mentioned limit in terms of TB. 🤷‍♂️

    Thanked by 1Ympker
  • NeoonNeoon Community Contributor, Veteran
    edited August 2019

    Well, its about 2 years ago, not that I remember that they had the 300Mbit thingy before. However, you can do it 2 ways, either you route "abusers" crappier like Online announced, cheap machines will be routed the cheapest way or you throttle them.

    No idea how servDiscount handles it now, I suppose they went away from the throttling idea.

    Thanked by 1Falzo
  • MikePTMikePT Moderator, Patron Provider, Veteran

    @DA_Mark said:
    We'd be glad to sponsor this if any LET provider wants to chime in and make an offer. @MasonR is a giver, not a taker. Please, someone get him operational within his deadline of tomorrow and we'll foot the bill. Premium guys welcome. No budget cap as long as it's reasonable.

    His call on who he chooses.

    ❤️❤️❤️

  • bacloudbacloud Member, Patron Provider

    60TB = around 200Mbps monthly usage. 1Gbps = 300TB+. Budget - $5. Doe's anyone now where I can buy unlimited dedicated 1Gbps for $25/month, 10Gbps for $250 or 100Gbps for just $2500?

    Selling VPS to OP means losing money :)

    Thanked by 1MasonR
  • ehabehab Member

    @bacloud said:

    off course

    Thanked by 2simonindia MasonR
  • @bacloud said:
    60TB = around 200Mbps monthly usage. 1Gbps = 300TB+. Budget - $5. Doe's anyone now where I can buy unlimited dedicated 1Gbps for $25/month, 10Gbps for $250 or 100Gbps for just $2500?

    Selling VPS to OP means losing money :)

    Yes, that's why I chimed in asked for LET hosts to bring their offers (no budget limits) and we'll sponsor it.

    If all else fails we'll still take care of @MasonR but I'd rather support a LET provider

    Thanked by 2Ouji MasonR
  • bacloud said: 60TB = around 200Mbps monthly usage. 1Gbps = 300TB+. Budget - $5.

    The 60TB of google drive space for $5/m or whatever is an even more entertaining aspect of this. That's actual hard drives though I'm happy to see Google resources going to a useful purpose like LET pron collections instead of surveillance ;). The bandwidth by comparison is no big deal, it's just some wire in a data center that's likely already connected to google through peering.

    Thanked by 2vimalware MasonR
  • Sorry I am late.
    @MasonR I can do this in Germany if it is not too late.

    Thanked by 2ehab MasonR
  • Please note that if you're moving 60TB of data, you're using 120TB of traffic, so that's 200mbit up AND 200mbit down.

    Three netcup vpsses would do it though.

    Thanked by 1MasonR
  • @HostDoc said:
    Sorry I am late.
    @MasonR I can do this in Germany if it is not too late.

    Doesn't have to be within OP's price range. We'll sponsor this at any (fair) price. Most important is that he can for-sure get the speed he needs for data transfer. Just want some offers posted so he can choose the one he prefers. :)

    Thanked by 2MasonR HostDoc
  • MasonRMasonR Community Contributor

    Thanks all for the interest! I've gone with an offer from @MikeA (ExtraVM). I already have services with him, so I know I'm in good hands, and he had some spare resources that I could tap into for this. I appreciate all the responses and offers, you guys are awesome!

    @DA_Mark said:
    Doesn't have to be within OP's price range. We'll sponsor this at any (fair) price. Most important is that he can for-sure get the speed he needs for data transfer. Just want some offers posted so he can choose the one he prefers. :)

    I've already had a chance to test with the ExtraVM VPS and it hits the speeds without any issues, so everything is up and running. Seems like it is a good fit. The bill was small enough that I took care of it for this month. I really appreciate the offer to sponsor this, if you feel so inclined you can talk to him about taking care of the additional 1 or 2 months it'll take for this to complete :) Thanks again, Mark!

  • MasonRMasonR Community Contributor

    @teamacc said:
    Please note that if you're moving 60TB of data, you're using 120TB of traffic, so that's 200mbit up AND 200mbit down.

    Three netcup vpsses would do it though.

    It's ~75mbit up and ~75mbit down (inc. tcp and google api overhead) so it doesn't breach the 750GB/day upload limit. Basically I can move about 23TB/month, so it'll take about 3 months at that rate no matter where it's done

  • HostDocHostDoc Member
    edited August 2019

    @DA_Mark said:

    @HostDoc said:
    Sorry I am late.
    @MasonR I can do this in Germany if it is not too late.

    Doesn't have to be within OP's price range. We'll sponsor this at any (fair) price. Most important is that he can for-sure get the speed he needs for data transfer. Just want some offers posted so he can choose the one he prefers. :)

    VZ Type: KVM
    Number of Cores: 1 [Fair Share (25%)] @4.0Ghz
    RAM: = 512MB
    Disk Space: 5GB
    Disk Type: SSD
    Bandwidth: 120TB
    Port Speed: =200 Mbps (Throttled)
    DDoS Protection: Yes
    Number of IPs: 1 IPv4
    Location: Germany
    Budget: = £4.00/m |$4.91/mo
    Billing period: Monthly

    Our Germany traffic does not use much bandwidth meaning you are safe with the allocation.
    It will be throttled to 200mbps on a node that has a 1gbps port.

    @DA_Mark You are truely a good guy.

    EDIT2: I was typing while @MasonR responded. Oops.

    Thanked by 3MasonR Ouji ITLabs
  • intovpsintovps Member, Host Rep

    I'm glad you found a solution. I'm curious if Google will let you use 150 Mbit/s constantly.

    And btw, doesn't this work with share + make a copy between the two Drive accounts?

  • MasonRMasonR Community Contributor

    @intovps said:
    I'm glad you found a solution. I'm curious if Google will let you use 150 Mbit/s constantly.

    Shouldn't be an issue. The daily upload limits are the only restrictions (for now). I've done something similar as this, albeit with not as much data

    And btw, doesn't this work with share + make a copy between the two Drive accounts?

    It does, but that process is still subject to the daily limit and it's much more manually intensive than just running a single command in a terminal window :P

    Thanked by 1intovps
  • MasonRMasonR Community Contributor

    Also, huge shout out and thank you to @DA_Mark for his generosity and help with this. You're a great person :)

    Thanked by 2ITLabs Ympker
  • group hug

    Thanked by 2MasonR vimalware
  • tester4tester4 Member
    edited January 2020

    This is a bump but if anyone else comes across this and needs to do something similar, Google Cloud itself is great. As GDrive is within the Google universe, bandwidth isn't charged and is unmetered gbit in both inbound AND outbound direction at the exact same time. I did this for the EXACT same reason as you (encrypting non-encrypted GDrive content) and worked a treat for 120TB of data.

    Thanked by 1MasonR
  • @tester4 said:
    This is a bump but if anyone else comes across this and needs to do something similar, Google Cloud itself is great. As GDrive is within the Google universe, bandwidth isn't charged and is unmetered gbit in both inbound AND outbound direction at the exact same time. I did this for the EXACT same reason as you (encrypting non-encrypted GDrive content) and worked a treat for 120TB of data.

    You can also use rclone with the --drive-server-side-across-configs flag. You'll need service accounts, but it's far faster than transferring through an ephemeral host. I've seen server side copy speeds of +-25GB/s

    Thanked by 2MasonR l2o88j
  • Cool. First mini necro of 2020. But the updates are informative

  • MasonRMasonR Community Contributor

    @tester4 said:
    This is a bump but if anyone else comes across this and needs to do something similar, Google Cloud itself is great. As GDrive is within the Google universe, bandwidth isn't charged and is unmetered gbit in both inbound AND outbound direction at the exact same time. I did this for the EXACT same reason as you (encrypting non-encrypted GDrive content) and worked a treat for 120TB of data.

    Yeah, could totally have done it on Google Cloud. For some reason a voice inside told me to not to trust gcloud since their bandwidth charges are outrageous, but I probably would have been fine.

    @td512 said:
    You can also use rclone with the --drive-server-side-across-configs flag. You'll need service accounts, but it's far faster than transferring through an ephemeral host. I've seen server side copy speeds of +-25GB/s

    That doesn't bypass the 750GB/day upload limit, right? Ultimately doesn't matter if you're doing the transfer all day (@ 75mbit/s) or for a short duration (@ 25gb/s) if you have a ton of data to move as it'll still take the same amount of time.

    @vyas11 said:
    Cool. First mini necro of 2020. But the updates are informative

    Yeah, not really a big deal as it's all relevant to the topic :)

    And now that this is bumped, I just want to give a big thanks to @MikeA for being beyond accomodating for this request. Worked like a charm. It did end up taking just over 3 months IIRC.

    Thanked by 1vyas11
  • @MasonR said:

    @tester4 said:
    This is a bump but if anyone else comes across this and needs to do something similar, Google Cloud itself is great. As GDrive is within the Google universe, bandwidth isn't charged and is unmetered gbit in both inbound AND outbound direction at the exact same time. I did this for the EXACT same reason as you (encrypting non-encrypted GDrive content) and worked a treat for 120TB of data.

    Yeah, could totally have done it on Google Cloud. For some reason a voice inside told me to not to trust gcloud since their bandwidth charges are outrageous, but I probably would have been fine.

    @td512 said:
    You can also use rclone with the --drive-server-side-across-configs flag. You'll need service accounts, but it's far faster than transferring through an ephemeral host. I've seen server side copy speeds of +-25GB/s

    That doesn't bypass the 750GB/day upload limit, right? Ultimately doesn't matter if you're doing the transfer all day (@ 75mbit/s) or for a short duration (@ 25gb/s) if you have a ton of data to move as it'll still take the same amount of time.

    @vyas11 said:
    Cool. First mini necro of 2020. But the updates are informative

    Yeah, not really a big deal as it's all relevant to the topic :)

    And now that this is bumped, I just want to give a big thanks to @MikeA for being beyond accomodating for this request. Worked like a charm. It did end up taking just over 3 months IIRC.

    The 750GB upload limit was problematic for me as well till I found out you could use team drive which had 750GB per ACCOUNT :open_mouth:

  • @MasonR said:

    @td512 said:
    You can also use rclone with the --drive-server-side-across-configs flag. You'll need service accounts, but it's far faster than transferring through an ephemeral host. I've seen server side copy speeds of +-25GB/s

    While true, that's why you use service accounts. then queue up a script that switches between them after 700GB. it reduced the time required for us to move 30TB from months to hours

    Thanked by 1MasonR
  • MasonRMasonR Community Contributor

    @tester4 said:
    The 750GB upload limit was problematic for me as well till I found out you could use team drive which had 750GB per ACCOUNT :open_mouth:

    Yeah, I was aware of that but didn't want to go through all the hassle. Just wanted to set and forget. Checked it a couple times a month to make sure it's progressing as expected. I didn't need the data to be moved over right away anyways, so I was fine with waiting.

  • MasonRMasonR Community Contributor

    @td512 said:

    @MasonR said:

    @td512 said:
    You can also use rclone with the --drive-server-side-across-configs flag. You'll need service accounts, but it's far faster than transferring through an ephemeral host. I've seen server side copy speeds of +-25GB/s

    While true, that's why you use service accounts. then queue up a script that switches between them after 700GB. it reduced the time required for us to move 30TB from months to hours

    Nice! I might play around with this soon so I am in better shape for next time.

  • @td512 said:

    While true, that's why you use service accounts. then queue up a script that switches between them after 700GB. it reduced the time required for us to move 30TB from months to hours

    Can this work on regular GSuite accounts?

Sign In or Register to comment.