All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
rsync 3.2.3 vs 3.1.2 benchmarks!
Folks may know I'm an avid user and tester of zstd compression and I've now integrated xxHash into my backup script routines. So when I totally missed the memo that rsync 3.2 added support for faster checksum hashing via xxHash and added lz4 and zstd compression support, I had to build a custom rsync 3.2.3 RPM for CentOS 7 and benchmark compared it to native CentOS 7's rsync 3.1.2 and the resulted gains with rsync 3.2.3 are awesome https://community.centminmod.com/threads/custom-rsync-3-2-3-rpm-builds-with-xxhash-zstd-support-for-centos-7.20768/ !
Knowing how many folks have multiple servers here and that they need to transfer data amongst those servers, I thought this would be of interest. So enjoy!
rsync 3.2.3
rsync --version
rsync version 3.2.3 protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, hardlink-specials, symlinks, IPv6, atimes,
batchfiles, inplace, append, ACLs, xattrs, optional protect-args, iconv,
symtimes, prealloc, stop-at, no crtimes
Optimizations:
SIMD, asm, openssl-crypto
Checksum list:
xxh128 xxh3 xxh64 (xxhash) md5 md4 none
Compress list:
zstd lz4 zlibx zlib none
rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.
rsync 3.1.2
rsync --version
rsync version 3.1.2 protocol version 31
Copyright (C) 1996-2015 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, iconv, symtimes, prealloc
rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
are welcome to redistribute it under certain conditions. See the GNU
General Public Licence for details.
FYI, I plan to add Rsync 3.2+ support to my Centmin Mod LEMP stack after doing more testing too
Comments
That is actually quite nice. I am looking forward to this speeding up several scripts....
Quite the good find. Thank you for sharing
I've read your test on your forum yesterday. My first question was just that, when would you add backup scripts to centminmod. lol.
Keep it up! You are the only reason that I still use CentOS. lol. All the servers which I have wordpress on have to be CentOS.
It looks like rsync 3.2 adds few optimizations and compression by default. rsync 3.1 - none. Did you compiled both versions from source or just used standard latest rpm's in version branch?
You're welcome
There's is one for MySQL database backups for Centmin Mod Premium members which also has S3, ftp and mobile push notification support https://community.centminmod.com/threads/dbbackup-sh-quick-mysql-database-backups-for-centmin-mod-stack.4573/ But I hear you and many folks that have asked regarding full backups!
It's work in progress as I've never been happy with the 4+ different backup scripts I've written - they'd be too technical for end users to use and understand. So need to get it right. The latest iteration is underway as part of Centmin Mod's full data transfer routine to another Centmin Mod server I am developing and testing which can transfer compressed data at near network/disk line rates maxing your servers out at 200-300MB/s (this was even before discovering Rsync 3.2.3 so can imagine things being faster!).
Some parts of it and the vision for them are previewed here and here. Part of backup routines will be to support remote Amazon S3 compatible storage as well.
I was excited to upgrade, but it turns out I'm already using rsync 3.2 on Debian. Haha
curious what's output for
rsync --version
?This is from a system running Debian bullseye (testing):
On my systems that run Buster, I sometimes use buster-backports, which has 3.2.2 rather than 3.2.3: https://packages.debian.org/rsync
nice has the full complement of optimizations, checksum and compression algorithms needed for that performance boost
@eva2000 Will rsync automatically use the best available hashing and compression algorithms (assuming the sender and client are both using a modern version), or do I need to explicitly opt in?
rsync 3.2.3 will default to xxh128 and zstd if no additional parameters are passed provided rsync 3.2.3 is built with zstd and xxhash support
How did you get that transfer speed figure? From rsync itself?
which one from rsync native output or from my scripted benchmarks ? scripted benchmarks one is sent bytes/transfer time
manual FTW https://man7.org/linux/man-pages/man1/rsync.1.html
So, we should see this hit the distro repo's between 2022 and 2026?
It's already in Debian buster-backports.
Yeah depends on distro some have rsync 3.2.3 some don't. See https://pkgs.org/download/rsync looks like Debian sid/bulleyes do as does Fedora Rawhide/33, Alpine Edge, Arch Linux and Ubuntu 20.10.
CentOS 8 is still 3.1.3 while CentOS 7 3.1.2. That's the beauty of Centmin Mod LEMP stack - I develop and build it so I do not have to wait for any distro upstream updates if I want to so build my own rsync 3.2.3 RPM binary and I can optionally insert bug/security patches into Centmin Mod's Nginx, PHP-FPM routines. Example patch fixing PHP <=7.2 for malformed communication packet errors introduced by stricter MariaDB packet validation.
As to rsync 3.2.3 I also did some compression level tests for rsync 3.1.2 zlib 1-6 and rsync 3.2.3 zstd -100 to 6 and lz4 1 to 6 at https://community.centminmod.com/threads/custom-rsync-3-2-3-rpm-builds-with-xxhash-zstd-support-for-centos-7.20768/#post-87368
Interesting rsync 3.2.3 zstd -29 compression level can match speed and compression ratio of lz4 at level 6 defaults
Had to investigate compression levels in rsync due to a possible bug in rsync 3.2.3 zlib compression as it might not be working properly https://github.com/WayneD/rsync/issues/123
Thank You eva for sharing. Good to know this.
You're welcome, did more benchmarks without pre-compressed log files in the mix due to rsync 3.2.3 zlib bug. Also added zstd fast negative compression levels up to -8000 yes rsync zstd has up to -131072 to 22 compression levels https://community.centminmod.com/threads/custom-rsync-3-2-3-rpm-builds-with-xxhash-zstd-support-for-centos-7.20768/#post-87375
At rsync 3.2.3 zstd -8000, transfer speed is almost as fast as no compressed rsync 3.2.3 and is faster than no compressed rsync 3.1.2 - 277MB/s vs 284MB/s ! Compared to rsync 3.1.2 compressed at 6.9MB/s !
Been putting Tar 1.32 and Rsync 3.2.3 to use with zstd compression and xxhash checksum algorithm speed up boots for file transfers in the real world and the results have been bloody awesome https://blog.centminmod.com/2021/01/30/2214/fast-tar-and-rsync-transfer-speed-for-linux-backups-using-zstd-compression/
Caught to the attention of CentOS 8 Stream maintainer too so maybe we might see Rsync 3.2.3 backported to RPM distros too
Pretty sure everyone on LET needs to move data between servers at one time or another, so here's to more folks switching to Tar 1.32 and Rsync 3.2.3 ^_^