Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Fastest way to transfer 1 TB of data between two dedicated servers? Maybe sync job too?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Fastest way to transfer 1 TB of data between two dedicated servers? Maybe sync job too?

aFriendaFriend Member

If someone has 2 dedicated servers (no private network), what is the fastest way to transport batches of data that can range between 10 GB to 600GB each? So far I use rsync over ssh. Any faster options?
Also, any helpful scripts, etc to set this up as a daily batch sync job? Im not an expert at setting up VPN, etc.. not sure if some sort of tunneling can help here.

Comments

  • raindog308raindog308 Administrator, Veteran

    said: So far I use rsync over ssh. Any faster options?

    In theory, normal rsync (i.e., port 873 - not over ssh) would be faster because it's not encrypted and there isn't the ssh protocol overhead.

    However (a) it's not encrypted, and (b) modern CPUs are fast enough that you'd be hard pressed to observe the difference. rsync does have its own access control (IP) and password auth - I use it for backups so the master can only pull read-only data.

    Fundamentally, you're moving blocks over the network. Since the network is the slowest point and you can't change it...ftp, rsync, scp, etc. are all going to have similar times.

    A couple things you could do:

    (a) compress on the fly or compress before sending
    (b) it's faster to transfer one big tar file than a million small files (less metadata)

    Thanked by 1aFriend
  • jarjar Patron Provider, Top Host, Veteran
    edited May 2014

    Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. –Andrew Tanenbaum

  • bobbybobby Member

    I was faced with this a while ago and did some testing.
    NFS performed best, no doubt.
    Note1: No encryption.
    Note2: Use iptables. Its wide open without. Google "securing nfs".

    NFS over openvpn was slow. SSHFS just as, if not faster.

    Good luck.

    Thanked by 1aFriend
  • Streaming with LZO.

    Thanked by 1aFriend
  • TarZZ92TarZZ92 Member

    ftp, btsync?

    Thanked by 1aFriend
  • There's always netcat, but I don't think you're going to see any meaningful difference, no matter how you do it.

    Thanked by 1aFriend
  • SilvengaSilvenga Member
    edited May 2014

    Btsync is probably more efficient than rsync - plus the built in encryption, hash checking, and less overhead because of UDP.

    Thanked by 1aFriend
  • raindog308raindog308 Administrator, Veteran

    Is that 10GB-600GB all new data? If so, a sync protocol vs. a transfer protocol doesn't buy you anything.

    If you have, say, 10TB of data and 10-600GB at random places changes on a daily basis, then a sync protocol could be very valuable.

    Thanked by 1aFriend
  • trexostrexos Member

    Does rsync over ssh include hash checking?

  • SilvengaSilvenga Member
    edited May 2014

    trexos said: Does rsync over ssh include hash checking?

    Rsync does not perform differently using SSH - so yes. I believe it uses MD4 for full file hashing. It might also perform a rolling hash check during the transfer.

    On the other hand BtSync will do a SHA-1 on each piece (1 MB?) and then a SHA-1 on the whole file.

    MD4 has statistical hash collisions so it should not be used for verification, but it can be used to prevent corruption - and MD4 is much faster than SHA-1. That said, I wonder what the side by side speed of Rsync when compared with Btsync is. Would be interesting to test.

    Thanked by 2trexos aFriend
  • raindog308raindog308 Administrator, Veteran

    Considering that CPU operations are likely several orders of magnitude wall-clock-wise than network operations, they're probably irrelevant. Even if one sync protocol was 2x the speed of another, that advantage will be insignificant if the network send/receive is 99.9% of the time.

  • Swapping the hard drives take 2 minutes :)

  • amhoabamhoab Member

    I can usually max out the link when using www.slac.stanford.edu/~abh/bbcp/

  • NeoonNeoon Community Contributor, Veteran

    @forthcloud yay did you made-up the teleportation?

    Just in the case its not in the same DC.

Sign In or Register to comment.