New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
How To: Securely Share Storage using NFS
I just wrote a basic guide on how to use WireGuard, NFS and optionally cachefilesd for local caching of NFS files, but this site keeps giving me a broken Cloudflare CAPTCHA when I try to post it:
So I posted it to my own blog instead: https://d.sb/2020/12/nfs-howto
Seeing as a bunch of people here have probably gotten storage VPSes during Black Friday and Christmas sales and are thinking of ways to use the storage space, I thought you'd find it useful
Comments
Good tutorial.
Do you have how to backup vps to storage vps with easy?
Thanks @Daniel15
@youandri For backups I use Borgbackup and Backupninja. Both are available in Debian (
apt install borgbackup backupninja
). I haven't written a tutorial for it, but here's some very rough unedited notes from a Google Doc I use when setting up my own servers:On backup server (storage server that holds the backups), add user and create backup dir:
(where
syd01
is the name of the origin server)On origin (server you want to back up), generate SSH key as root. Do NOT set a passphrase:
Get the public key from ~/.ssh/id_ed25519.pub. On backup server, create
authorized_keys
for user using this public key, and restrict it to only allow Borgbackup (no regular SSH access using this key):On origin, create Borg repo and backup key:
Enter a randomly generated password when prompted. Store
key
AND the password in a safe place (eg in LastPass) as you need BOTH to decrypt the backups.Then you can configure Backupninja using
ninjahelper
, and modify / etc/backupninja.conf to set when the backups run, where it sends emails, and if it sends emails on success or only on failure.Nice write-up! I'm curious if this is resilient against a sometimes shitty network?
By default it'll keep retrying indefinitely, so in theory it should handle unreliable networks. If a client tries to read from or write to the NFS share, and the NFS server is down, it'll block until the server is back up (or you give up and kill the process ).
You can change it to stop retrying after a period of time (the
soft
option) but that can result in data loss. For example, if a write times out, that write will be lost. I would recommend always usinghard
mode, which is the default. You can see the nfs(5) man page for more detailsWhat is the advantage of using NFS over mounting SFTP as a network drive or using SMB with user authentication?
Awesome! Thanks for clarifying that, even though my question was quite vague you answered the concerns I had about dropouts etc.
I've been using rclone and sftp and the dropouts really take the jam out of my donut.
Thanks for the great write-up! How long does it keep the files in cache? Does it get removed after some time?
The cache keeps growing until the free disk space or inodes drops below a particular limit, at which point it'll start removing the least recently used files from the cache. To change the max cache size, you can modify the cachefilesd config, or use a separate partition just for the cache.
Edit: Nevermind
Thanks @Daniel15 for the tutorial.
I was wondering what are the other options that does a similar thing? this way is the fastest way?
There's a few other options:
SSHFS is one of the most popular. It just uses SSH, so no extra configuration is required on the server side, as long as SFTP is enabled on your server (it's enabled by default). It's fairly stable but can have reliability issues if you try to read/write a lot of files in parallel (SFTP really isn't designed for that use case). SSHFS does not have any active contributors so any bugs you encounter are likely to not get fixed unless you fix them yourself
SMB is popular too. It's the protocol Windows uses for file sharing. You have to install "Samba" to use it. It does support username and password authentication out-of-the-box. I think SMB3 supports encryption but I'm not 100% sure. I haven't touched Samba for many many years now so I'm not sure how it compares to NFS these days. It used to have a reputation for being insecure back in the Windows 98 days, but newer versions are better.
SSHFS and Samba are both apps you need to install and run, whereas NFS is built in to the Linux kernel (the bits you install are just the commands to administer it). In theory, kernel code is faster and more efficient than "userland" (non-kernel) code.
Here's an article I just found with a performance comparison: https://blog.ja-ke.tech/2019/08/27/nas-performance-sshfs-nfs-smb.html
Thanks for the details @Daniel15
What about Gluster over Wireguard without replication? Would that be comparable to NFS over Wireguard at all?
I see this: https://www.howtoforge.com/creating-an-nfs-like-standalone-storage-server-with-glusterfs-3.2.x-on-ubuntu-12.10
(outdated and doesn't secure it with Wireguard though)
wouldn't using wireguard also route all your internet traffic through the tunnel? making this not so efficient?
Thank you for explaining the cache part. The rest I had already mastered on my own. Runs great so far.
No. In the tutorial I only use a single IP in the
AllowedIPs
in each WireGuard peer (eg.10.123.0.1/32
), meaning just that one IP is routed to that node via the VPN. This is usually called a "peer-to-peer" or "point-to-point" VPN. This is the way that WireGuard works by default out-of-the-box. No packet forwarding is taking place and it's just peers directly communicating with other peers.To route all your traffic through WireGuard (which is often what people think of when they hear "VPN"), you'd need to allow all IPs (
0.0.0.0/0
), and also add someiptables
rules to do the routing. I don't do that.I haven't tried Gluster so I'm not too sure about that.
thanks @Daniel15 , for a good tutorial,. sorry for noob question. after establishing the connection with WireGuard, how could I test the up/down speed between the server and client. is it just by downloading files or there are other ways? thanks
You'd want to use a program like
iperf3
.On a basic level (Quick speed test without tuning options) this would just be
iperf3 -s -B 10.123.0.1
on one end andiperf3 -c 10.123.0.1
on the other. Of course replace the IP;-s
is the server end and-c
is the client. I took the IP from the tutorial in question here.great. thanks
@Daniel15 in your experience have you found wireguard to be good at reconnecting if the connection drops etc?
You'll be glad to hear aside from occasional handshakes, WireGuard does not really have much of a concept of "reconnecting". When you send a packet through a created WireGuard interface it is simply encrypted with the private key and sent to its appropriate endpoint regardless of current state. At a simplified level this is all that WireGuard is doing, encrypting packets over a virtual network. You define who you can send packets to and who can send you packets, over which ports and using which encryption keys.
In its bare form it provides unparalleled performance across shaky networks, functioning the same as any other (UDP) internet protocol traffic would. Using WireGuard on a mobile device with a mobile network feels like it doesn't even have it active, connection is as shaky as your location and supporting applications.
So the handling of a situation where your network is dropping frequently would be down to the other protocol in use, for this example it would be NFS but in other cases could be SSH, HTTP, etc.
You can configure it with a persistent keepalive handshake, but that's mostly to avoid confusing some NATs/firewalls and doesn't do much except keep the connection appearing alive even when its idling (So the NAT doesnt assume you're done with that endpoint and cut you off).
(Feel free to correct me if I'm wrong, this is just what I've come to learn over reading and use)
I'd use iperf3 like @Erisa said. Note that it won't be as fast as an unencrypted connection, as encryption isn't free (it needs some CPU power). However, WireGuard should be faster than some other VPN systems like OpenVPN.
WireGuard doesn't really 'connect' nor resend lost packets nor handle reordering packets that arrive out of order, as it's UDP. UDP just sends packets over the internet without checking if they arrive properly or in the right order.
Stuff like that is at a higher layer than WireGuard. When you perform TCP connections over the VPN (like HTTP requests, NFS connections, etc), those connections handle the reconnections and retransmissions, as part of TCP. WireGuard doesn't have to do it, as the protocols being tunneled over the VPN already do it. So, basically what @Erisa said.
This is also why it's usually a bad idea to use a TCP connection for a VPN. It's sometimes useful to bypass proxies (like at school or a workplace that blocks UDP traffic), but there's much more overhead, as you're tunnelling TCP over TCP. The nature of TCP means that all the packets must arrive in exactly the same order as they're sent, with nothing dropped, which makes it a lot slower.
Wiregurd uses less memory and CPU than other protocols VPN like openvpn/pptp/l2tp
thanks @Daniel15 , by the way. anyone having issue with etc/fstab config. it seems not working with ubuntu18. i tried it multiple times and its not attaching after reboot. however, using manual "mount -a" it is working.
Maybe the data you entered not complete, the format should be like this:
thanks..i will try.. but i am using the same config as mentioned in @Daniel15 blog. like below
To automatically mount the directory on boot, modify /etc/fstab:
10.123.0.1:/data/storage /mnt/data nfs4 auto,vers=4.2
Here's the exact line I use in
/etc/fstab
on one of my systems, and it works fine:That's strange... I think booting is supposed to do the same thing as
mount -a
. I'm not sure what the problem could be here.If you're using HostHatch, make sure you enable "jumbo frames" by setting the MTU to 9000, for best performance. On Debian, edit
/etc/network/interfaces
and addmtu 9000
under the relevant section (I think both their public and internal networks support jumbo frames).You can double-check the MTU values by checking
/sys/class/net/<device>/mtu
:(as per https://lists.zx2c4.com/pipermail/wireguard/2017-December/002201.html, WireGuard has overhead of 80 bytes per packet, hence the slightly lower MTU for the WireGuard interface).
I was getting 380 Mb/s using WireGuard between two VPSes (measured using
iperf3
), but after bumping the MTU from 1500 to 9000 I'm now getting 1.4 Gb/s over the VPN.Nice tutorial, thanks for sharing !
Thanks for sharing