New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
How to check progress of File Transfer Using TAR
Hello,
I am using TAR to transfer file from one server to another. The command itself is working properly but I am not able to check the real time progress like rsync.
Can anyone here modify the command to show progress ?
tar -cz /home/test/ | ssh 127.0.0.1 'tar -xz -C /home/'
Thanks.
Comments
from
man tar-v, --verbose verbosely list files processedSo, you might just need to add that.
thanks a lot man, this at-least show a type of progress, which list the files as they are being written on remote server ...
tar -cz test | ssh 127.0.0.1 'tar -xz -C /home/' -vdo you have any recommendation to make the transfer faster ... I have good hardware and network on both server, only thing I don't have is disk space on source server ...
Thank you again.
after testing several different stuff: fastest rsync command for me:
rsync -rtXx --numeric-ids --progress -e "ssh -T -c arcfour -o Compression=no -x" <user>@<ip>:/<source-path>/ /<dest.path>/run this on receiving server.
Are you writing across ssh to a server on the same network or something outside? Removing compression would probably be quite a bit faster, but my guess is the limit will be the network it's going across. I'd suggest leaving out -v if you don't actually need to watch the output, there's no point of it since it isn't a way to measure transfer progress.
Edit: Yeah, rsync would probably be better and has more functionality anyways.
Take a look at
pv:tar cf - test | pv -petarbs $(( $( find test -type f -printf '%s + ' ) 0 )) | ssh 127.0.0.1 'tar xf - -C /home/'This won't be 100% acurate as far as percent/ETA are concerned since the find part is merely guessing the size of the tar file but it's pretty close. If you have/want to use compression you will have to adjust the calculation or just remove the size argument from pv (it will not be able to display percent/ETA then) though.
@emre needed to remove the cipher thing ... rest working command, but it doesn't really so much different than the existing rsync command I was using ... still that slowness
ssh by default does compression so can try emitting that from tar:
tar -c /home/test/ | ssh 127.0.0.1 'tar -x -C /home/'You can also put pv in-between to see the transfer speed in real time.
tar -c /home/test/ | pv | ssh 127.0.0.1 'tar -x -C /home/'You can use the "lsof" command to see what files are open on a system, and use the "-o 0" option to show the file offset in decimal. You will want to pipe the output through grep to eliminate a lot of extraneous stuff. That gives a good indication of how far the tarring has gotten. See the man page to understand the output format.
Tar can use
--totals=USR1, which will show the total bytes written and throughput when triggered (killed) with SIGUSR1:killall -USR1 tarbut personally I would just eyeball it withpv -s <whatever size you expect>between the pipes -- or the way @mksh described it.You may also want to take a look at
rsync.Where exactly is the bottleneck? I imagine the source HDD has to do quite a bit of seeking if there are a lot of possibly fragmented files (since you mentioned the drive may be full?).
If the network is fully utilized you probably need to resort to compression and if you have more than one core I strongly recommend installing the multi-threaded drop-in replacements for bzip, gzip and xz: https://gist.github.com/vmp32k/16030297e77c36366df53e7ef3e91720