Best way to copy large files from server to server (Thats not Rsync)
New on LowEndTalk? Please read our 'Community Rules' by clicking on it in the right menu!

Best way to copy large files from server to server (Thats not Rsync)

HuntersPadHuntersPad Member
edited May 2015 in Help

Any suggestions? Have been trying for 2 days now moving my site via FTP

Old Provider>Windows RDP Server> to new Provider (Only 250GB) so doing it in bunches.

Well my freaking Windows RDP provider keeps going down and I havent even transferred near half of my stuff.

Rsync is TOO slow Takes about 5+min per file at times

zipping takes about 5min to zip/tar a 55MB file (lol)

Any suggestions? or any Linux VPS with Ghome and filezilla installed I could borrow for 24 hours?

Thanks!

HostMantis Shared Hosting: UK, Canada, Germany, Singapore, Sydney, New York Sign up Here!


Check out my site! hunterspad.com Destruction videos, tech, random, etc

«1

Comments

  • joepie91joepie91 Member, Provider
  • @joepie91 said:
    rsync isn't slow. The problem is likely elsewhere.

    Rsync runs fine, except for when it transfers large files 2GB+ which makes up for 90% of the files.

    HostMantis Shared Hosting: UK, Canada, Germany, Singapore, Sydney, New York Sign up Here!


    Check out my site! hunterspad.com Destruction videos, tech, random, etc

  • joepie91joepie91 Member, Provider

    hyelton said: Rsync runs fine, except for when it transfers large files 2GB+ which makes up for 90% of the files.

    I use rsync on a daily basis, and it does 2GB+ files just fine. Again, the problem is likely elsewhere. Are you tunneling rsync over SSH?

  • AnthonySmithAnthonySmith Top Provider

    robocopy is pretty robust, or you could simply attach the new VM as iSCSI storage to the old one set the iSCSI drive as a backup point, do a full windows BM backup then do a restore on the new server, you get pretty significant compression that way too.

    Thanked by 1HyperSpeed
  • edited May 2015

    You can try multithreading with aria2...

    aria2c -x8 -i listofurls.txt

    (8 threads example)

    But honestly, the problem (like @joepie91 said) is likely elsewhere...

    Thanked by 2netomx ehab
  • @joepie91 said:

    Yeah, and the old provider system load is off the charts just doing FTP, but at least FTP is way faster, its just my problem that the RDP provider keeps going down so FTP is becoming not an option anymore/

    HostMantis Shared Hosting: UK, Canada, Germany, Singapore, Sydney, New York Sign up Here!


    Check out my site! hunterspad.com Destruction videos, tech, random, etc

  • I just know I need about 700GB transfered. Any VPS provider willing to provide a high storage VPS for 24 hours?

    HostMantis Shared Hosting: UK, Canada, Germany, Singapore, Sydney, New York Sign up Here!


    Check out my site! hunterspad.com Destruction videos, tech, random, etc

  • AnthonySmithAnthonySmith Top Provider

    Are they both KVM?

    You could always boot them both in to sysrescuecd or equivalent and just dd the whole disk image over, then boot it on the other end.

  • joepie91joepie91 Member, Provider

    hyelton said: Yeah, and the old provider system load is off the charts just doing FTP, but at least FTP is way faster, its just my problem that the RDP provider keeps going down so FTP is becoming not an option anymore/

    Any kind of file transfer over SSH is going to be slow, due to the way SSH is designed. Try running a plain rsync server (you'll need to configure rsyncd for this, it's not hard), and you'll find that speeds are much, much better.

  • AnthonySmithAnthonySmith Top Provider

    hyelton said: I just know I need about 700GB transfered.

    Holy crap! where did you get a VM with over 700 GB storage?

  • @AnthonySmith said:
    Holy crap! where did you get a VM with over 700 GB storage?

    That is a Dedi on my old provider its 2TB, But Im moving things to a VPS.

    HostMantis Shared Hosting: UK, Canada, Germany, Singapore, Sydney, New York Sign up Here!


    Check out my site! hunterspad.com Destruction videos, tech, random, etc

  • @AnthonySmith said:
    Are they both KVM?

    You could always boot them both in to sysrescuecd or equivalent and just dd the whole disk image over, then boot it on the other end.

    Old provider is a Dedicated Server and new is OpenVZ.

    HostMantis Shared Hosting: UK, Canada, Germany, Singapore, Sydney, New York Sign up Here!


    Check out my site! hunterspad.com Destruction videos, tech, random, etc

  • AnthonySmithAnthonySmith Top Provider

    Oh right your going from windows on a dedi to a VPS on Linux, I understand now :)

    Not sure I understand why you need to extra 700GB VPS though if the dedi keeps going down that's not going to help if its the OpenVZ VPS going down then whats the point of moving there and who the hell is providing an OpenVZ VPS with 700GB storage lol?

    Anyway none of that helps you sorry I am just curious and a bit confused..

  • hyelton said: That is a Dedi on my old provider its 2TB, But Im moving things to a VPS.

    This sounds like a recipe for disaster. Does your VPS provider have sufficient bandwidth (in reality rather than their dreams) to support your site?

  • rokokrokok Member

    Bittorrent sync

    CEO of PT. Rokok Kopi Internet Tidur Tbk.

  • HuntersPadHuntersPad Member
    edited May 2015

    @MarkTurner said:

    @AnthyonySmith

    A little misunderstanding here.

    My Old Provider is Linux/CentOS.

    By RDP I mean a Windows Server That Im using in the middle to transfer the files from one server to another via FTP, which the provider of THAT server keeps going down.

    Copying files to the Windows Server from Old provider, and from Windows Server to new provider. Im bascially using the Windows server for its 500mbps connection to transfer.

    And my new provider/VPS its on a OVH server. So I`m sure theres plenty of Bandwidth.

    HostMantis Shared Hosting: UK, Canada, Germany, Singapore, Sydney, New York Sign up Here!


    Check out my site! hunterspad.com Destruction videos, tech, random, etc

  • Why can't you copy from one server to the other directly.

    Just shutdown the webserver on the old server temporarily.

  • @MarkTurner said:
    Why can't you copy from one server to the other directly.

    Just shutdown the webserver on the old server temporarily.

    How? What other ways are there? Rsync is way to slow.

    HostMantis Shared Hosting: UK, Canada, Germany, Singapore, Sydney, New York Sign up Here!


    Check out my site! hunterspad.com Destruction videos, tech, random, etc

  • rsync is fine, I have moved thousands of of multi gigabyte files with that.

    Thanked by 1FHR
  • @MarkTurner said:
    rsync is fine, I have moved thousands of of multi gigabyte files with that.

    rsync coming from the old provider takes several min per file, With FTP I can transfer about 6 files in a few min.

    HostMantis Shared Hosting: UK, Canada, Germany, Singapore, Sydney, New York Sign up Here!


    Check out my site! hunterspad.com Destruction videos, tech, random, etc

  • If I knew I wouldnt muck anything up, I`d install Ghome GUI and tightvnc on my old provider.

    HostMantis Shared Hosting: UK, Canada, Germany, Singapore, Sydney, New York Sign up Here!


    Check out my site! hunterspad.com Destruction videos, tech, random, etc

  • AnthonySmithAnthonySmith Top Provider

    well in that case you are doing it twice essentially adding who knows what latency in between and honestly for no good reason.

    you can just rsync the files directly from one to the other, if you need help with the commands to do that over cli just say as that would be far more simple than what you are doing, you could even just scp the files over e.g.

    if rsync over command line worrys you then just use the most simple method.

    scp /home/myoldfiles/* [email protected]:/home/mynewfiles/
    

    enter your password for the new server and just let it run which will be as fast if not faster than ftp.

  • raindog308raindog308 Moderator

    hyelton said: Rsync is way to slow.

    Paste your rsync command. I suspect there's an -e ssh in there that you're not telling us about. Unless you're running rsyncd on one side, you're running rsync-over-ssh which will give you ssh-like speeds.

    All other things being equal, speed is going to be, fastest to slowest:

    (1) FTP (fastest, not encrypted)
    (2) rsync
    (3) ssh/scp/sftp (encrypts)

    Of these, only rsync is resumable.

    joepie91 said: Any kind of file transfer over SSH is going to be slow, due to the way SSH is designed. Try running a plain rsync server (you'll need to configure rsyncd for this, it's not hard), and you'll find that speeds are much, much better.

    Listen to @joepie91

    My Advice: VPS Advice

    For LET support, please click here.

  • @AnthonySmith said:
    well in that case you are doing it twice essentially adding who knows what latency in between and honestly for no good reason.

    you can just rsync the files directly from one to the other, if you need help with the commands to do that over cli just say as that would be far more simple than what you are doing, you could even just scp the files over e.g.

    if rsync over command line worrys you then just use the most simple method.

    > scp /home/myoldfiles/* [email protected]:/home/mynewfiles/
    > 

    enter your password for the new server and just let it run which will be as fast if not faster than ftp.

    Thanks, How can I do spaces in the folder name?

    EX: /LG G3/Stock doesnt work says its invalid but its clearly there.

    HostMantis Shared Hosting: UK, Canada, Germany, Singapore, Sydney, New York Sign up Here!


    Check out my site! hunterspad.com Destruction videos, tech, random, etc

  • sirmbhesirmbhe Member

    I manage large files using btsync, works flawlessly. But, I did that on a dedicated server. I once try to do that using VPS, and my provider message me about me abusing the node's io. :D

    Thanked by 2rokok agonyzt
  • I just need a VPS with 200-300GB storage for 48 hours and it will all be resolved lol.

    HostMantis Shared Hosting: UK, Canada, Germany, Singapore, Sydney, New York Sign up Here!


    Check out my site! hunterspad.com Destruction videos, tech, random, etc

  • tridinebandimtridinebandim Member
    edited May 2015

    hyelton said: /LG G3/Stock

    try this/LG\ G3/Stock

    Use Tab to auto complete path names

    Hi!

  • You should have got a concierge service from your new provider. This could be done so much simpler.

  • @hyelton said:
    I just need a VPS with 200-300GB storage for 48 hours and it will all be resolved lol.

    Vultr storage plan?__

    Reality does hurt!

  • jcalebjcaleb Moderator

    In the old days, we use sneakernet for fast and reliable large file transfers.

    btw, how about you tar then split your entire folder. then transfer to the other side

  • TACServersTACServers Member, Provider

    @hyelton - I'll let you have a Linux or Windows VM on a gigabit network with 500GB of storage. For 48 hours. Sound like a deal?

  • The issue you are seeing with how long files take is likely the overhead from encryption, as the encryption will take a good bit of CPU and on servers with low resources this can cause a slow down. This is also why FTP (assuming you are using it without SSL) would be much faster, as you are removing the encryption thus allowing more CPU for just file transfer. Honestly, if you are worried about how long its taking FTP may be your best bet followed by using rsync without encryption (without SSH). Also, any type of compression will also slow things down, as once again you will be relying on CPU time to compress things. If you are not concerned with someone sniffing your data, then finding a method to transfer which doesn't use encryption will be significantly faster than with.

    Cheers!

    Have an Allwinner H3 device? Want Android? Check out H3Droid! | Lichee Pi Zero - The 6$ SBC | #SYSarm - Get It!
    18+ Years IT Experience in Linux/Windows Hosting and Administration

  • NomadNomad Member

    If the servers are close, you can always try NFS as well.

    NameSilo.com coupons: CheapDoms or Discounted
    - Send Everything My Way

  • Looks like either the source or destination server does not have enough processing power and rsync over SSH is slow because of that. Try using rsyncd. If you want something simpler, then try playing with the compression and encryption options. For a decent guide, see https://gist.github.com/KartikTalwar/4393116 or http://janaksingh.com/blog/faster-rsync-over-ssh-using-arcfour-encryption-146

    If you insist on using FTP, then try wget with the 'mirror' option to directly copy the files to the destination server from the source server. See this http://serverfault.com/questions/25199/using-wget-to-recursively-download-whole-ftp-directories

    Using a Windows server in between is probably not the best option. If you insist on doing that, then try Vultr or any other provider which allows hourly billing and supports Windows.

  • joepie91joepie91 Member, Provider
    edited May 2015

    raindog308 said: All other things being equal, speed is going to be, fastest to slowest:

    (1) FTP (fastest, not encrypted) (2) rsync (3) ssh/scp/sftp (encrypts)

    TheLinuxBug said: The issue you are seeing with how long files take is likely the overhead from encryption, as the encryption will take a good bit of CPU and on servers with low resources this can cause a slow down.

    nikhil500 said: Looks like either the source or destination server does not have enough processing power and rsync over SSH is slow because of that.

    I'd actually suspect rsync to be faster than FTP, due to the lack of roundtrips (FTP is very bad with this).

    Also, SSH isn't slow because it encrypts, it's slow because of the way it buffers up data, and how that is affected by latency. I forgot the specifics, though. There's a patch around somewhere that fixes that. Certainly, it's not the encryption causing it.

    Thanked by 1raindog308
  • I'll echo the others' comments. 1) rsync isn't slow, there must be something else wrong. 2) try btsync.

    Signatures are to identify who I am. I'm me. Who the hell are you?

  • vfusevfuse Member, Provider

    You can try lftp, supports sftp and can use multiple connections.

    NIXStats monitoring service (servers, http(s), ICMP, tcp ports) - monitoring 13,000+ servers - Uptime Report - API Docs

  • BuyAdsBuyAds Member

    Why you just dont compress the files and wget them?

    Buy Site Ads - Advertising Platform

  • @BuyAds said:
    Why you just dont compress the files and wget them?

    Well its about 800GB of files, AND my old server takes about 5min to compress 55MB so theres a problem with that lol.

    HostMantis Shared Hosting: UK, Canada, Germany, Singapore, Sydney, New York Sign up Here!


    Check out my site! hunterspad.com Destruction videos, tech, random, etc

  • Source: Install a simple webserver and turn directory listings on.

    Target: wget -r --no-parent http://mysite.com/mydirectory/

    Signatures are to identify who I am. I'm me. Who the hell are you?

  • perennateperennate Member, Provider
    edited May 2015

    What rsync command are you using? People ask but you didn't say. You can disable compression by not including the -z flag.

    Also check the CPU usage when rsync is running, see if it's maxing out the CPU, if so then that's probably because of the encryption overhead.

    Installing webserver and using wget is always easy option.

  • It already has a Webserver I havent tried that command.

    HostMantis Shared Hosting: UK, Canada, Germany, Singapore, Sydney, New York Sign up Here!


    Check out my site! hunterspad.com Destruction videos, tech, random, etc

  • @rajprakash said:
    Source: Install a simple webserver and turn directory listings on.

    Target: wget -r --no-parent http://mysite.com/mydirectory/

    How do I turn Directory listings on? I currently use a PHP directory listing, but going directly to it is 403.

    HostMantis Shared Hosting: UK, Canada, Germany, Singapore, Sydney, New York Sign up Here!


    Check out my site! hunterspad.com Destruction videos, tech, random, etc

  • Figured it out

    HostMantis Shared Hosting: UK, Canada, Germany, Singapore, Sydney, New York Sign up Here!


    Check out my site! hunterspad.com Destruction videos, tech, random, etc

  • @rajprakash said:
    Source: Install a simple webserver and turn directory listings on.

    Target: wget -r --no-parent http://mysite.com/mydirectory/

    THanks it works!, But I just choose /Downloads but it copies everything thats before the folder as well.

    HostMantis Shared Hosting: UK, Canada, Germany, Singapore, Sydney, New York Sign up Here!


    Check out my site! hunterspad.com Destruction videos, tech, random, etc

  • oijpghjighojioijpghjighoji Member
    edited May 2015
  • getvpsgetvps Member

    Maybe packing everything or using containers (like truecrypt) and after downloading via http/ftp? .. less size,more protection :)

  • twaintwain Member

    If you really want a gui-like option you could install mc on the src side and FTP/SFTP the files over the FISH protocol. Midnight Commander is available from almost every linux dist standard repo and has an ncurses gui. Not sure if FISH supports resuming or not..

  • sc754sc754 Member

    I mount my backup servers hard drive to my webserver using sshfs, then just cp the file. Never had any problems with it

  • edited March 13

    What rsync command are you using? People ask but you didn't say. You can disable compression by not including the -z flag.

    Also check the CPU usage when rsync is running, see if it's maxing out the CPU, if so then that's probably because of the encryption overhead.

    Installing webserver and using wget is always easy option..you can also use third party software to rysnc large file...i am using [some shitty bullshittery tool] for rysnc large number of files.

This discussion has been closed.