rsync 3.2.3 vs 3.1.2 benchmarks!

eva2000

Folks may know I'm an avid user and tester of zstd compression and I've now integrated xxHash into my backup script routines. So when I totally missed the memo that rsync 3.2 added support for faster checksum hashing via xxHash and added lz4 and zstd compression support, I had to build a custom rsync 3.2.3 RPM for CentOS 7 and benchmark compared it to native CentOS 7's rsync 3.1.2 and the resulted gains with rsync 3.2.3 are awesome https://community.centminmod.com/threads/custom-rsync-3-2-3-rpm-builds-with-xxhash-zstd-support-for-centos-7.20768/ !

Knowing how many folks have multiple servers here and that they need to transfer data amongst those servers, I thought this would be of interest. So enjoy!

rsync 3.2.3

rsync --version
rsync  version 3.2.3  protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, hardlink-specials, symlinks, IPv6, atimes,
    batchfiles, inplace, append, ACLs, xattrs, optional protect-args, iconv,
    symtimes, prealloc, stop-at, no crtimes
Optimizations:
    SIMD, asm, openssl-crypto
Checksum list:
    xxh128 xxh3 xxh64 (xxhash) md5 md4 none
Compress list:
    zstd lz4 zlibx zlib none

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

rsync 3.1.2

rsync --version
rsync  version 3.1.2  protocol version 31
Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

FYI, I plan to add Rsync 3.2+ support to my Centmin Mod LEMP stack after doing more testing too

SGraf

That is actually quite nice. I am looking forward to this speeding up several scripts....

Quite the good find. Thank you for sharing

kend

FYI, I plan to add Rsync 3.2+ support to my Centmin Mod LEMP stack after doing more testing too

I've read your test on your forum yesterday. My first question was just that, when would you add backup scripts to centminmod. lol.

Keep it up! You are the only reason that I still use CentOS. lol. All the servers which I have wordpress on have to be CentOS.

Levi

It looks like rsync 3.2 adds few optimizations and compression by default. rsync 3.1 - none. Did you compiled both versions from source or just used standard latest rpm's in version branch?

eva2000

@SGraf said: Quite the good find. Thank you for sharing

You're welcome

@kend said: My first question was just that, when would you add backup scripts to centminmod. lol.

There's is one for MySQL database backups for Centmin Mod Premium members which also has S3, ftp and mobile push notification support https://community.centminmod.com/threads/dbbackup-sh-quick-mysql-database-backups-for-centmin-mod-stack.4573/ But I hear you and many folks that have asked regarding full backups!

It's work in progress as I've never been happy with the 4+ different backup scripts I've written - they'd be too technical for end users to use and understand. So need to get it right. The latest iteration is underway as part of Centmin Mod's full data transfer routine to another Centmin Mod server I am developing and testing which can transfer compressed data at near network/disk line rates maxing your servers out at 200-300MB/s (this was even before discovering Rsync 3.2.3 so can imagine things being faster!).

Some parts of it and the vision for them are previewed here and here. Part of backup routines will be to support remote Amazon S3 compatible storage as well.

@LTniger said: It looks like rsync 3.2 adds few optimizations and compression by default. rsync 3.1 - none. Did you compiled both versions from source or just used standard latest rpm's in version branch?

• rsync 3.1.2 is standard CentOS 7 distro provided RPM
• rsync 3.2.3 is my own custom built RPM binary

Daniel15

I was excited to upgrade, but it turns out I'm already using rsync 3.2 on Debian. Haha

eva2000

@Daniel15 said: I was excited to upgrade, but it turns out I'm already using rsync 3.2 on Debian. Haha

curious what's output for rsync --version ?

Daniel15

@eva2000 said:

@Daniel15 said: I was excited to upgrade, but it turns out I'm already using rsync 3.2 on Debian. Haha

curious what's output for rsync --version ?

This is from a system running Debian bullseye (testing):

rsync  version 3.2.3  protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, hardlink-specials, symlinks, IPv6, atimes,
    batchfiles, inplace, append, ACLs, xattrs, optional protect-args, iconv,
    symtimes, prealloc, stop-at, no crtimes
Optimizations:
    SIMD, asm, openssl-crypto
Checksum list:
    xxh128 xxh3 xxh64 (xxhash) md5 md4 none
Compress list:
    zstd lz4 zlibx zlib none

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

On my systems that run Buster, I sometimes use buster-backports, which has 3.2.2 rather than 3.2.3: https://packages.debian.org/rsync

eva2000

@Daniel15 said: This is from a system running Debian bullseye (testing):

nice has the full complement of optimizations, checksum and compression algorithms needed for that performance boost

Daniel15

@eva2000 Will rsync automatically use the best available hashing and compression algorithms (assuming the sender and client are both using a modern version), or do I need to explicitly opt in?

eva2000

rsync 3.2.3 will default to xxh128 and zstd if no additional parameters are passed provided rsync 3.2.3 is built with zstd and xxhash support

time rsync -avzin --debug=NSTR /home/zcat-test/ /home/zcat-test-dst/
Client negotiated checksum: xxh128
Client negotiated compress: zstd (level 3)
sending incremental file list
.d..t...... ./
>f+++++++++ access_log_20180428-234724.log.gz
>f+++++++++ access_log_20180428-234724.log.zst
>f+++++++++ access_log_20180429-005239.log.gz
>f+++++++++ access_log_20180429-005239.log.zst
>f+++++++++ access_log_20180429-012648.log.gz
>f+++++++++ access_log_20180429-012648.log.zst

sent 237 bytes  received 37 bytes  548.00 bytes/sec
total size is 263,394,826  speedup is 961,294.99 (DRY RUN)

real    0m0.044s
user    0m0.000s
sys     0m0.003s

greattomeetyou

How did you get that transfer speed figure? From rsync itself?

eva2000

@greattomeetyou said: How did you get that transfer speed figure? From rsync itself?

which one from rsync native output or from my scripted benchmarks ? scripted benchmarks one is sent bytes/transfer time

eva2000

@Daniel15 said: Will rsync automatically use the best available hashing and compression algorithms (assuming the sender and client are both using a modern version), or do I need to explicitly opt in?

manual FTW https://man7.org/linux/man-pages/man1/rsync.1.html

The "auto" option is the default, where rsync bases its
algorithm choice on a negotiation between the client and the
server as follows:

          When both sides of the transfer are at least 3.2.0, rsync
          chooses the first algorithm in the client's list of choices
          that is also in the server's list of choices.  If no common
          checksum choice is found, rsync exits with an error.  If the
          remote rsync is too old to support checksum negotiation, a
          value is chosen based on the protocol version (which chooses
          between MD5 and various flavors of MD4 based on protocol age).

          The default order can be customized by setting the environment
          variable RSYNC_CHECKSUM_LIST to a space-separated list of
          acceptable checksum names.  If the string contains a "&"
          character, it is separated into the "client string & server
          string", otherwise the same string applies to both.  If the
          string (or string portion) contains no non-whitespace
          characters, the default checksum list is used.  This method
          does not allow you to specify the transfer checksum separately
          from the pre-transfer checksum, and it discards "auto" and all
          unknown checksum names.  A list with only invalid names
          results in a failed negotiation.

          The use of the --checksum-choice option overrides this
          environment list.

TimboJones

So, we should see this hit the distro repo's between 2022 and 2026?

Daniel15

@TimboJones said:
So, we should see this hit the distro repo's between 2022 and 2026?

It's already in Debian buster-backports.

eva2000

@TimboJones said: So, we should see this hit the distro repo's between 2022 and 2026?

Yeah depends on distro some have rsync 3.2.3 some don't. See https://pkgs.org/download/rsync looks like Debian sid/bulleyes do as does Fedora Rawhide/33, Alpine Edge, Arch Linux and Ubuntu 20.10.

CentOS 8 is still 3.1.3 while CentOS 7 3.1.2. That's the beauty of Centmin Mod LEMP stack - I develop and build it so I do not have to wait for any distro upstream updates if I want to so build my own rsync 3.2.3 RPM binary and I can optionally insert bug/security patches into Centmin Mod's Nginx, PHP-FPM routines. Example patch fixing PHP <=7.2 for malformed communication packet errors introduced by stricter MariaDB packet validation.

As to rsync 3.2.3 I also did some compression level tests for rsync 3.1.2 zlib 1-6 and rsync 3.2.3 zstd -100 to 6 and lz4 1 to 6 at https://community.centminmod.com/threads/custom-rsync-3-2-3-rpm-builds-with-xxhash-zstd-support-for-centos-7.20768/#post-87368

Interesting rsync 3.2.3 zstd -29 compression level can match speed and compression ratio of lz4 at level 6 defaults

• Compression ratio wise, rsync 3.2.3 zstd with xxh128 checksum needs fast -29 compression level to end up sending around the same amount of bytes as lz4 compression. At zstd -29 speed is ~143.8MB/s while lz4 is around 143.5MB/s at lz4 default level 6 compression.
• rsync 3.2.3 zstd -100 compression level is interesting as the speed was much faster at ~526.6MB/s but zstd -99 drops back to ~193.2MB/s.

Had to investigate compression levels in rsync due to a possible bug in rsync 3.2.3 zlib compression as it might not be working properly https://github.com/WayneD/rsync/issues/123

MGarbis

Thank You eva for sharing. Good to know this.

eva2000

@MGarbis said:
Thank You eva for sharing. Good to know this.

You're welcome, did more benchmarks without pre-compressed log files in the mix due to rsync 3.2.3 zlib bug. Also added zstd fast negative compression levels up to -8000 yes rsync zstd has up to -131072 to 22 compression levels https://community.centminmod.com/threads/custom-rsync-3-2-3-rpm-builds-with-xxhash-zstd-support-for-centos-7.20768/#post-87375

At rsync 3.2.3 zstd -8000, transfer speed is almost as fast as no compressed rsync 3.2.3 and is faster than no compressed rsync 3.1.2 - 277MB/s vs 284MB/s ! Compared to rsync 3.1.2 compressed at 6.9MB/s !

/usr/bin/rsync -avzi --stats --log-file=/home/rsync-benchmarks/rsyncbench-rsync-native-compressed-051220-224524.log /home/rsync-test-no-precomp/ /home/rsync-test-no-precomp-dst/
[rsync 3.1.2 native compressed] files: 2,842 reg: 2,841, dir: 1
[rsync 3.1.2 native compressed] total bytes: 510556971 sent bytes: 55648190 (6553198.59 per second)
[rsync 3.1.2 native compressed] transfer speed (MB/s): 6.8922 speedup: 9.17
[rsync 3.1.2 native compressed] real: 7.70s user: 7.66s sys: 0.50s cpu: 106% maxmem: 2332 KB cswaits: 5855

/usr/bin/rsync -avi --stats --log-file=/home/rsync-benchmarks/rsyncbench-rsync-native-no-compress-051220-224524.log /home/rsync-test-no-precomp/ /home/rsync-test-no-precomp-dst/
[rsync 3.1.2 native no-compress] files: 2,842 reg: 2,841, dir: 1
[rsync 3.1.2 native no-compress] total bytes: 510556971 sent bytes: 510977977 (204412790.00 per second)
[rsync 3.1.2 native no-compress] transfer speed (MB/s): 213.7309 speedup: 1.00
[rsync 3.1.2 native no-compress] real: 2.28s user: 1.66s sys: 0.79s cpu: 107% maxmem: 1988 KB cswaits: 16267

/usr/local/bin/rsync -avzi --stats --cc xxh128 --zc zstd --zl -8000 --debug=NSTR --log-file=/home/rsync-benchmarks/rsyncbench-xxh128-zstd-051220-233002.log /home/rsync-test-no-precomp/ /home/rsync-test-no-precomp-dst/
[rsync 3.2.3 xxh128-zstd--8000 files: 2,842 reg: 2,841, dir: 1
[rsync 3.2.3 xxh128-zstd--8000 total bytes: 510556971 sent bytes: 502664954 (335145968.00 per second)
[rsync 3.2.3 xxh128-zstd--8000 transfer speed (MB/s): 277.0974 speedup: 1.02
[rsync 3.2.3 xxh128-zstd--8000 real: 1.73s user: 0.34s sys: 0.62s cpu: 56% maxmem: 3888 KB cswaits: 13102

/usr/local/bin/rsync -avzi --stats --cc xxh128 --zc none --debug=NSTR --log-file=/home/rsync-benchmarks/rsyncbench-xxh128-none-051220-233220.log /home/rsync-test-no-precomp/ /home/rsync-test-no-precomp-dst/
[rsync 3.2.3 xxh128-none] files: 2,842 reg: 2,841, dir: 1
[rsync 3.2.3 xxh128-none] total bytes: 510556971 sent bytes: 510977988 (340687990.67 per second)
[rsync 3.2.3 xxh128-none] transfer speed (MB/s): 284.9746 speedup: 1.00
[rsync 3.2.3 xxh128-none] real: 1.71s user: 0.18s sys: 0.69s cpu: 51% maxmem: 2744 KB cswaits: 19841

eva2000

Been putting Tar 1.32 and Rsync 3.2.3 to use with zstd compression and xxhash checksum algorithm speed up boots for file transfers in the real world and the results have been bloody awesome https://blog.centminmod.com/2021/01/30/2214/fast-tar-and-rsync-transfer-speed-for-linux-backups-using-zstd-compression/

Caught to the attention of CentOS 8 Stream maintainer too so maybe we might see Rsync 3.2.3 backported to RPM distros too

https://twitter.com/Arrfab/status/1355798789899104258

Pretty sure everyone on LET needs to move data between servers at one time or another, so here's to more folks switching to Tar 1.32 and Rsync 3.2.3 ^_^