Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


What is better FTP or Rsync for 200GB mp3 dirctory
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

What is better FTP or Rsync for 200GB mp3 dirctory

I would like to back up a 200GB mp3 directory from one server to another. Which would be faster to do it all in one clip, FTP or Rsync?

Comments

  • Anything but FTP, unless you are talking about SFTP or FTPS.

  • oops, I meant Sftp, not Ftp.

  • singsingsingsing Member
    edited September 2015

    By default, rsync skips file transfer if size and modtime match on the destination, making it take no more CPU than the other options in simple cases. In complex cases it will save some transfer at the cost of CPU usage.

  • I would suggest rsync as you can resume from failure.
    lftp mirror may achieve the same goal.

  • bookstack said: I would suggest rsync as you can resume from failure. lftp mirror may achieve the same goal.

    That's the real issue. For a one time transfer the speed is not important as it would be 200GB regardless.

  • If you have a lot of files, don't use ftp. ftp sends each file in a new connection.

  • just tar it all up and use axel. would be done by now.

  • rsync strongly recommended over ftp. Syncing big amount of data is very likely to fail at some point.

  • I've had upto 4x tx rate improvement over plain rsync (over ssh) by using lftp(over sftp/ssh). It was an emergency move on storage plan expiry.

    Admittedly, that was for large multi GB files where lftp's multi-part downloading helped get past the source provider/upstream's annoying per-connection caps. Going from 6MB/s to 22MB/sec definitely helps for transferring 100s of GBs.

    If you have enough space on source disk, tar up the whole payload and try lftp with 3-6 parallel connections. Else, rsync with resume is appropriate.

  • rsync is great :)

  • rsync 100%

    Thanked by 14n0nx
  • I went with Rsync

    Thanked by 14n0nx
  • Rsync. I just transferred 2.3 gb worth of files both by rysnc and sftp. Rsync completed the task in about 90 minutes, whereas sftp had been running for 13 hours and still not complete. Not to mention rsync has incremental backup so if a file already exists at the target location, it will be skipped, saving bandwidth and time.

    Thanked by 14n0nx
  • singsingsingsing Member
    edited September 2015

    thagoat said: Rsync completed the task in about 90 minutes, whereas sftp had been running for 13 hours and still not complete.

    Let me guess, you were transferring a whole bunch of small files? The difference comes from the fact that sftp waits for each file to complete before starting the next. This limits performance to a few RTT (= round-trip-time) per file, regardless of your ultimate bandwidth transfer capability. Piping tar through ssh avoids this problem if you really don't want to use rsync.

    Thanked by 14n0nx
Sign In or Register to comment.