Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Best way to copy large files from server to server (Thats not Rsync)
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Best way to copy large files from server to server (Thats not Rsync)

HuntersPadHuntersPad Member
edited May 2015 in Help

Any suggestions? Have been trying for 2 days now moving my site via FTP

Old Provider>Windows RDP Server> to new Provider (Only 250GB) so doing it in bunches.

Well my freaking Windows RDP provider keeps going down and I havent even transferred near half of my stuff.

Rsync is TOO slow Takes about 5+min per file at times

zipping takes about 5min to zip/tar a 55MB file (lol)

Any suggestions? or any Linux VPS with Ghome and filezilla installed I could borrow for 24 hours?

Thanks!

«1

Comments

  • joepie91joepie91 Member, Patron Provider

    rsync isn't slow. The problem is likely elsewhere.

  • @joepie91 said:
    rsync isn't slow. The problem is likely elsewhere.

    Rsync runs fine, except for when it transfers large files 2GB+ which makes up for 90% of the files.

  • joepie91joepie91 Member, Patron Provider

    hyelton said: Rsync runs fine, except for when it transfers large files 2GB+ which makes up for 90% of the files.

    I use rsync on a daily basis, and it does 2GB+ files just fine. Again, the problem is likely elsewhere. Are you tunneling rsync over SSH?

  • AnthonySmithAnthonySmith Member, Patron Provider

    robocopy is pretty robust, or you could simply attach the new VM as iSCSI storage to the old one set the iSCSI drive as a backup point, do a full windows BM backup then do a restore on the new server, you get pretty significant compression that way too.

    Thanked by 1HyperSpeed
  • edited May 2015

    You can try multithreading with aria2...

    aria2c -x8 -i listofurls.txt

    (8 threads example)

    But honestly, the problem (like @joepie91 said) is likely elsewhere...

    Thanked by 2netomx ehab
  • @joepie91 said:

    Yeah, and the old provider system load is off the charts just doing FTP, but at least FTP is way faster, its just my problem that the RDP provider keeps going down so FTP is becoming not an option anymore/

  • I just know I need about 700GB transfered. Any VPS provider willing to provide a high storage VPS for 24 hours?

  • AnthonySmithAnthonySmith Member, Patron Provider

    Are they both KVM?

    You could always boot them both in to sysrescuecd or equivalent and just dd the whole disk image over, then boot it on the other end.

  • joepie91joepie91 Member, Patron Provider

    hyelton said: Yeah, and the old provider system load is off the charts just doing FTP, but at least FTP is way faster, its just my problem that the RDP provider keeps going down so FTP is becoming not an option anymore/

    Any kind of file transfer over SSH is going to be slow, due to the way SSH is designed. Try running a plain rsync server (you'll need to configure rsyncd for this, it's not hard), and you'll find that speeds are much, much better.

  • AnthonySmithAnthonySmith Member, Patron Provider

    hyelton said: I just know I need about 700GB transfered.

    Holy crap! where did you get a VM with over 700 GB storage?

  • @AnthonySmith said:
    Holy crap! where did you get a VM with over 700 GB storage?

    That is a Dedi on my old provider its 2TB, But Im moving things to a VPS.

  • @AnthonySmith said:
    Are they both KVM?

    You could always boot them both in to sysrescuecd or equivalent and just dd the whole disk image over, then boot it on the other end.

    Old provider is a Dedicated Server and new is OpenVZ.

  • AnthonySmithAnthonySmith Member, Patron Provider

    Oh right your going from windows on a dedi to a VPS on Linux, I understand now :)

    Not sure I understand why you need to extra 700GB VPS though if the dedi keeps going down that's not going to help if its the OpenVZ VPS going down then whats the point of moving there and who the hell is providing an OpenVZ VPS with 700GB storage lol?

    Anyway none of that helps you sorry I am just curious and a bit confused..

  • hyelton said: That is a Dedi on my old provider its 2TB, But Im moving things to a VPS.

    This sounds like a recipe for disaster. Does your VPS provider have sufficient bandwidth (in reality rather than their dreams) to support your site?

  • rokokrokok Member

    Bittorrent sync

  • HuntersPadHuntersPad Member
    edited May 2015

    @MarkTurner said:

    @AnthyonySmith

    A little misunderstanding here.

    My Old Provider is Linux/CentOS.

    By RDP I mean a Windows Server That Im using in the middle to transfer the files from one server to another via FTP, which the provider of THAT server keeps going down.

    Copying files to the Windows Server from Old provider, and from Windows Server to new provider. Im bascially using the Windows server for its 500mbps connection to transfer.

    And my new provider/VPS its on a OVH server. So I`m sure theres plenty of Bandwidth.

  • Why can't you copy from one server to the other directly.

    Just shutdown the webserver on the old server temporarily.

  • @MarkTurner said:
    Why can't you copy from one server to the other directly.

    Just shutdown the webserver on the old server temporarily.

    How? What other ways are there? Rsync is way to slow.

  • rsync is fine, I have moved thousands of of multi gigabyte files with that.

    Thanked by 1FHR
  • @MarkTurner said:
    rsync is fine, I have moved thousands of of multi gigabyte files with that.

    rsync coming from the old provider takes several min per file, With FTP I can transfer about 6 files in a few min.

  • If I knew I wouldnt muck anything up, I`d install Ghome GUI and tightvnc on my old provider.

  • AnthonySmithAnthonySmith Member, Patron Provider

    well in that case you are doing it twice essentially adding who knows what latency in between and honestly for no good reason.

    you can just rsync the files directly from one to the other, if you need help with the commands to do that over cli just say as that would be far more simple than what you are doing, you could even just scp the files over e.g.

    if rsync over command line worrys you then just use the most simple method.

    scp /home/myoldfiles/* [email protected]:/home/mynewfiles/
    

    enter your password for the new server and just let it run which will be as fast if not faster than ftp.

  • raindog308raindog308 Administrator, Veteran

    hyelton said: Rsync is way to slow.

    Paste your rsync command. I suspect there's an -e ssh in there that you're not telling us about. Unless you're running rsyncd on one side, you're running rsync-over-ssh which will give you ssh-like speeds.

    All other things being equal, speed is going to be, fastest to slowest:

    (1) FTP (fastest, not encrypted)
    (2) rsync
    (3) ssh/scp/sftp (encrypts)

    Of these, only rsync is resumable.

    joepie91 said: Any kind of file transfer over SSH is going to be slow, due to the way SSH is designed. Try running a plain rsync server (you'll need to configure rsyncd for this, it's not hard), and you'll find that speeds are much, much better.

    Listen to @joepie91

  • @AnthonySmith said:
    well in that case you are doing it twice essentially adding who knows what latency in between and honestly for no good reason.

    you can just rsync the files directly from one to the other, if you need help with the commands to do that over cli just say as that would be far more simple than what you are doing, you could even just scp the files over e.g.

    if rsync over command line worrys you then just use the most simple method.

    > scp /home/myoldfiles/* [email protected]:/home/mynewfiles/
    > 

    enter your password for the new server and just let it run which will be as fast if not faster than ftp.

    Thanks, How can I do spaces in the folder name?

    EX: /LG G3/Stock doesnt work says its invalid but its clearly there.

  • sirmbhesirmbhe Member

    I manage large files using btsync, works flawlessly. But, I did that on a dedicated server. I once try to do that using VPS, and my provider message me about me abusing the node's io. :D

    Thanked by 2rokok agonyzt
  • I just need a VPS with 200-300GB storage for 48 hours and it will all be resolved lol.

  • tridinebandimtridinebandim Member
    edited May 2015

    hyelton said: /LG G3/Stock

    try this/LG\ G3/Stock

    Use Tab to auto complete path names

  • You should have got a concierge service from your new provider. This could be done so much simpler.

  • @hyelton said:
    I just need a VPS with 200-300GB storage for 48 hours and it will all be resolved lol.

    Vultr storage plan?__

  • jcalebjcaleb Member

    In the old days, we use sneakernet for fast and reliable large file transfers.

    btw, how about you tar then split your entire folder. then transfer to the other side

This discussion has been closed.