How To: Securely Share Storage using NFS

Daniel15 · January 1

I just wrote a basic guide on how to use WireGuard, NFS and optionally cachefilesd for local caching of NFS files, but this site keeps giving me a broken Cloudflare CAPTCHA when I try to post it:

So I posted it to my own blog instead: https://d.sb/2020/12/nfs-howto

Seeing as a bunch of people here have probably gotten storage VPSes during Black Friday and Christmas sales and are thinking of ways to use the storage space, I thought you'd find it useful

youandri · January 1

Good tutorial.

Do you have how to backup vps to storage vps with easy?

ferri · January 1

Thanks @Daniel15

Daniel15 · January 1

@youandri For backups I use Borgbackup and Backupninja. Both are available in Debian (apt install borgbackup backupninja). I haven't written a tutorial for it, but here's some very rough unedited notes from a Google Doc I use when setting up my own servers:

On backup server (storage server that holds the backups), add user and create backup dir:

adduser backup-syd01 --disabled-password
mkdir /backup/syd01
chown backup-syd01:backup-syd01 /backup/syd01
chmod 0700 /backup/syd01

(where syd01 is the name of the origin server)

On origin (server you want to back up), generate SSH key as root. Do NOT set a passphrase:

ssh-keygen -o -a 100 -t ed25519 -f ~/.ssh/id_ed25519 -C "[email protected]"

Get the public key from ~/.ssh/id_ed25519.pub. On backup server, create authorized_keys for user using this public key, and restrict it to only allow Borgbackup (no regular SSH access using this key):

command="borg serve --append-only --restrict-to-path /backup/syd01",restrict ssh-ed25519 AAAA........ [email protected]

On origin, create Borg repo and backup key:

borg init --encryption=keyfile-blake2 [email protected]:/backup/syd01
borg key export [email protected]:/backup/syd01 key
cat key

Enter a randomly generated password when prompted. Store key AND the password in a safe place (eg in LastPass) as you need BOTH to decrypt the backups.

Then you can configure Backupninja using ninjahelper, and modify / etc/backupninja.conf to set when the backups run, where it sends emails, and if it sends emails on success or only on failure.

its420somewhere · January 1

Nice write-up! I'm curious if this is resilient against a sometimes shitty network?

Daniel15 · January 1

@its420somewhere said:
Nice write-up! I'm curious if this is resilient against a sometimes shitty network?

By default it'll keep retrying indefinitely, so in theory it should handle unreliable networks. If a client tries to read from or write to the NFS share, and the NFS server is down, it'll block until the server is back up (or you give up and kill the process ).

You can change it to stop retrying after a period of time (the soft option) but that can result in data loss. For example, if a write times out, that write will be lost. I would recommend always using hard mode, which is the default. You can see the nfs(5) man page for more details

user123 · January 1

What is the advantage of using NFS over mounting SFTP as a network drive or using SMB with user authentication?

its420somewhere · January 1

@Daniel15 said:

@its420somewhere said:
Nice write-up! I'm curious if this is resilient against a sometimes shitty network?

By default it'll keep retrying indefinitely, so in theory it should handle unreliable networks. If a client tries to read from or write to the NFS share, and the NFS server is down, it'll block until the server is back up (or you give up and kill the process ).

You can change it to stop retrying after a period of time (the soft option) but that can result in data loss. For example, if a write times out, that write will be lost. I would recommend always using hard mode, which is the default. You can see the nfs(5) man page for more details

Awesome! Thanks for clarifying that, even though my question was quite vague you answered the concerns I had about dropouts etc.

I've been using rclone and sftp and the dropouts really take the jam out of my donut.

SP7 · January 1

Thanks for the great write-up! How long does it keep the files in cache? Does it get removed after some time?

Daniel15 · January 1

@SP7 said:
Thanks for the great write-up! How long does it keep the files in cache? Does it get removed after some time?

The cache keeps growing until the free disk space or inodes drops below a particular limit, at which point it'll start removing the least recently used files from the cache. To change the max cache size, you can modify the cachefilesd config, or use a separate partition just for the cache.

SP7 · January 1

Edit: Nevermind

Kassem · January 1

Thanks @Daniel15 for the tutorial.

I was wondering what are the other options that does a similar thing? this way is the fastest way?

Daniel15 · January 1

@Kassem said: I was wondering what are the other options that does a similar thing?

There's a few other options:

SSHFS is one of the most popular. It just uses SSH, so no extra configuration is required on the server side, as long as SFTP is enabled on your server (it's enabled by default). It's fairly stable but can have reliability issues if you try to read/write a lot of files in parallel (SFTP really isn't designed for that use case). SSHFS does not have any active contributors so any bugs you encounter are likely to not get fixed unless you fix them yourself

SMB is popular too. It's the protocol Windows uses for file sharing. You have to install "Samba" to use it. It does support username and password authentication out-of-the-box. I think SMB3 supports encryption but I'm not 100% sure. I haven't touched Samba for many many years now so I'm not sure how it compares to NFS these days. It used to have a reputation for being insecure back in the Windows 98 days, but newer versions are better.

SSHFS and Samba are both apps you need to install and run, whereas NFS is built in to the Linux kernel (the bits you install are just the commands to administer it). In theory, kernel code is faster and more efficient than "userland" (non-kernel) code.

Here's an article I just found with a performance comparison: https://blog.ja-ke.tech/2019/08/27/nas-performance-sshfs-nfs-smb.html

Kassem · January 2

Thanks for the details @Daniel15

What about Gluster over Wireguard without replication? Would that be comparable to NFS over Wireguard at all?

I see this: https://www.howtoforge.com/creating-an-nfs-like-standalone-storage-server-with-glusterfs-3.2.x-on-ubuntu-12.10

(outdated and doesn't secure it with Wireguard though)

jugganuts · January 2

wouldn't using wireguard also route all your internet traffic through the tunnel? making this not so efficient?

benj0x · January 2

Thank you for explaining the cache part. The rest I had already mastered on my own. Runs great so far.

Daniel15 · January 2

@jugganuts said:
wouldn't using wireguard also route all your internet traffic through the tunnel? making this not so efficient?

No. In the tutorial I only use a single IP in the AllowedIPs in each WireGuard peer (eg. 10.123.0.1/32), meaning just that one IP is routed to that node via the VPN. This is usually called a "peer-to-peer" or "point-to-point" VPN. This is the way that WireGuard works by default out-of-the-box. No packet forwarding is taking place and it's just peers directly communicating with other peers.

To route all your traffic through WireGuard (which is often what people think of when they hear "VPN"), you'd need to allow all IPs (0.0.0.0/0), and also add some iptables rules to do the routing. I don't do that.

@Kassem said: What about Gluster over Wireguard without replication? Would that be comparable to NFS over Wireguard at all?

I haven't tried Gluster so I'm not too sure about that.

ravenchad · January 3

@Daniel15 said:
I just wrote a basic guide on how to use WireGuard, NFS and optionally cachefilesd for local caching of NFS files, but this site keeps giving me a broken Cloudflare CAPTCHA when I try to post it:

So I posted it to my own blog instead: https://d.sb/2020/12/nfs-howto

Seeing as a bunch of people here have probably gotten storage VPSes during Black Friday and Christmas sales and are thinking of ways to use the storage space, I thought you'd find it useful

thanks @Daniel15 , for a good tutorial,. sorry for noob question. after establishing the connection with WireGuard, how could I test the up/down speed between the server and client. is it just by downloading files or there are other ways? thanks

Erisa · January 3

@ravenchad said: thanks @Daniel15 , for a good tutorial,. sorry for noob question. after establishing the connection with WireGuard, how could I test the up/down speed between the server and client? thanks

You'd want to use a program like iperf3.
On a basic level (Quick speed test without tuning options) this would just be iperf3 -s -B 10.123.0.1 on one end and iperf3 -c 10.123.0.1 on the other. Of course replace the IP; -s is the server end and -c is the client. I took the IP from the tutorial in question here.

ravenchad · January 3

@Erisa said:

@ravenchad said: thanks @Daniel15 , for a good tutorial,. sorry for noob question. after establishing the connection with WireGuard, how could I test the up/down speed between the server and client? thanks

You'd want to use a program like iperf3.
On a basic level (Quick speed test without tuning options) this would just be iperf3 -s -B 10.123.0.1 on one end and iperf3 -c 10.123.0.1 on the other. Of course replace the IP; -s is the server end and -c is the client. I took the IP from the tutorial in question here.

great. thanks

its420somewhere · January 3

@Daniel15 in your experience have you found wireguard to be good at reconnecting if the connection drops etc?

Erisa · January 3

@its420somewhere said:
@Daniel15 in your experience have you found wireguard to be good at reconnecting if the connection drops etc?

You'll be glad to hear aside from occasional handshakes, WireGuard does not really have much of a concept of "reconnecting". When you send a packet through a created WireGuard interface it is simply encrypted with the private key and sent to its appropriate endpoint regardless of current state. At a simplified level this is all that WireGuard is doing, encrypting packets over a virtual network. You define who you can send packets to and who can send you packets, over which ports and using which encryption keys.

In its bare form it provides unparalleled performance across shaky networks, functioning the same as any other (UDP) internet protocol traffic would. Using WireGuard on a mobile device with a mobile network feels like it doesn't even have it active, connection is as shaky as your location and supporting applications.
So the handling of a situation where your network is dropping frequently would be down to the other protocol in use, for this example it would be NFS but in other cases could be SSH, HTTP, etc.

You can configure it with a persistent keepalive handshake, but that's mostly to avoid confusing some NATs/firewalls and doesn't do much except keep the connection appearing alive even when its idling (So the NAT doesnt assume you're done with that endpoint and cut you off).

(Feel free to correct me if I'm wrong, this is just what I've come to learn over reading and use)

Daniel15 · January 4

@ravenchad said: after establishing the connection with WireGuard, how could I test the up/down speed between the server and client.

I'd use iperf3 like @Erisa said. Note that it won't be as fast as an unencrypted connection, as encryption isn't free (it needs some CPU power). However, WireGuard should be faster than some other VPN systems like OpenVPN.

@its420somewhere said: in your experience have you found wireguard to be good at reconnecting if the connection drops etc?

WireGuard doesn't really 'connect' nor resend lost packets nor handle reordering packets that arrive out of order, as it's UDP. UDP just sends packets over the internet without checking if they arrive properly or in the right order.

Stuff like that is at a higher layer than WireGuard. When you perform TCP connections over the VPN (like HTTP requests, NFS connections, etc), those connections handle the reconnections and retransmissions, as part of TCP. WireGuard doesn't have to do it, as the protocols being tunneled over the VPN already do it. So, basically what @Erisa said.

This is also why it's usually a bad idea to use a TCP connection for a VPN. It's sometimes useful to bypass proxies (like at school or a workplace that blocks UDP traffic), but there's much more overhead, as you're tunnelling TCP over TCP. The nature of TCP means that all the packets must arrive in exactly the same order as they're sent, with nothing dropped, which makes it a lot slower.

isunbejo · January 4

Wiregurd uses less memory and CPU than other protocols VPN like openvpn/pptp/l2tp

ravenchad · January 4

@Daniel15 said:

@ravenchad said: after establishing the connection with WireGuard, how could I test the up/down speed between the server and client.

I'd use iperf3 like @Erisa said. Note that it won't be as fast as an unencrypted connection, as encryption isn't free (it needs some CPU power). However, WireGuard should be faster than some other VPN systems like OpenVPN.

@its420somewhere said: in your experience have you found wireguard to be good at reconnecting if the connection drops etc?

WireGuard doesn't really 'connect' nor resend lost packets nor handle reordering packets that arrive out of order, as it's UDP. UDP just sends packets over the internet without checking if they arrive properly or in the right order.

Stuff like that is at a higher layer than WireGuard. When you perform TCP connections over the VPN (like HTTP requests, NFS connections, etc), those connections handle the reconnections and retransmissions, as part of TCP. WireGuard doesn't have to do it, as the protocols being tunneled over the VPN already do it. So, basically what @Erisa said.

This is also why it's usually a bad idea to use a TCP connection for a VPN. It's sometimes useful to bypass proxies (like at school or a workplace that blocks UDP traffic), but there's much more overhead, as you're tunnelling TCP over TCP. The nature of TCP means that all the packets must arrive in exactly the same order as they're sent, with nothing dropped, which makes it a lot slower.

thanks @Daniel15 , by the way. anyone having issue with etc/fstab config. it seems not working with ubuntu18. i tried it multiple times and its not attaching after reboot. however, using manual "mount -a" it is working.

chocolateshirt · January 4

@ravenchad said:

@Daniel15 said:

@ravenchad said: after establishing the connection with WireGuard, how could I test the up/down speed between the server and client.

I'd use iperf3 like @Erisa said. Note that it won't be as fast as an unencrypted connection, as encryption isn't free (it needs some CPU power). However, WireGuard should be faster than some other VPN systems like OpenVPN.

@its420somewhere said: in your experience have you found wireguard to be good at reconnecting if the connection drops etc?

WireGuard doesn't really 'connect' nor resend lost packets nor handle reordering packets that arrive out of order, as it's UDP. UDP just sends packets over the internet without checking if they arrive properly or in the right order.

Stuff like that is at a higher layer than WireGuard. When you perform TCP connections over the VPN (like HTTP requests, NFS connections, etc), those connections handle the reconnections and retransmissions, as part of TCP. WireGuard doesn't have to do it, as the protocols being tunneled over the VPN already do it. So, basically what @Erisa said.

This is also why it's usually a bad idea to use a TCP connection for a VPN. It's sometimes useful to bypass proxies (like at school or a workplace that blocks UDP traffic), but there's much more overhead, as you're tunnelling TCP over TCP. The nature of TCP means that all the packets must arrive in exactly the same order as they're sent, with nothing dropped, which makes it a lot slower.

thanks @Daniel15 , by the way. anyone having issue with etc/fstab config. it seems not working with ubuntu18. i tried it multiple times and its not attaching after reboot. however, using manual "mount -a" it is working.

Maybe the data you entered not complete, the format should be like this:

# <file system>     <dir>       <type>   <options>   <dump> <pass>
10.10.0.10:/backups /your/local/mount nfs      defaults    0       0

ravenchad · January 4

@chocolateshirt said:

@ravenchad said:

@Daniel15 said:

@ravenchad said: after establishing the connection with WireGuard, how could I test the up/down speed between the server and client.

I'd use iperf3 like @Erisa said. Note that it won't be as fast as an unencrypted connection, as encryption isn't free (it needs some CPU power). However, WireGuard should be faster than some other VPN systems like OpenVPN.

@its420somewhere said: in your experience have you found wireguard to be good at reconnecting if the connection drops etc?

WireGuard doesn't really 'connect' nor resend lost packets nor handle reordering packets that arrive out of order, as it's UDP. UDP just sends packets over the internet without checking if they arrive properly or in the right order.

Stuff like that is at a higher layer than WireGuard. When you perform TCP connections over the VPN (like HTTP requests, NFS connections, etc), those connections handle the reconnections and retransmissions, as part of TCP. WireGuard doesn't have to do it, as the protocols being tunneled over the VPN already do it. So, basically what @Erisa said.

This is also why it's usually a bad idea to use a TCP connection for a VPN. It's sometimes useful to bypass proxies (like at school or a workplace that blocks UDP traffic), but there's much more overhead, as you're tunnelling TCP over TCP. The nature of TCP means that all the packets must arrive in exactly the same order as they're sent, with nothing dropped, which makes it a lot slower.

thanks @Daniel15 , by the way. anyone having issue with etc/fstab config. it seems not working with ubuntu18. i tried it multiple times and its not attaching after reboot. however, using manual "mount -a" it is working.

Maybe the data you entered not complete, the format should be like this:
# <file system>     <dir>       <type>   <options>   <dump>   <pass>
10.10.0.10:/backups /your/local/mount nfs      defaults    0       0

thanks..i will try.. but i am using the same config as mentioned in @Daniel15 blog. like below

To automatically mount the directory on boot, modify /etc/fstab:

10.123.0.1:/data/storage /mnt/data nfs4 auto,vers=4.2

Daniel15 · January 4

Here's the exact line I use in /etc/fstab on one of my systems, and it works fine:

chi01.vpn.d.sb:/data/nfs/chi03 /mnt/data nfs4 auto,fsc,vers=4.2,async

@ravenchad said: it seems not working with ubuntu18. i tried it multiple times and its not attaching after reboot. however, using manual "mount -a" it is working.

That's strange... I think booting is supposed to do the same thing as mount -a. I'm not sure what the problem could be here.

Daniel15 · January 5

If you're using HostHatch, make sure you enable "jumbo frames" by setting the MTU to 9000, for best performance. On Debian, edit /etc/network/interfaces and add mtu 9000 under the relevant section (I think both their public and internal networks support jumbo frames).

You can double-check the MTU values by checking /sys/class/net/<device>/mtu:

[email protected]:~#  cat /sys/class/net/eth1/mtu
9000
[email protected]:~#  cat /sys/class/net/wg0/mtu
8920

(as per https://lists.zx2c4.com/pipermail/wireguard/2017-December/002201.html, WireGuard has overhead of 80 bytes per packet, hence the slightly lower MTU for the WireGuard interface).

I was getting 380 Mb/s using WireGuard between two VPSes (measured using iperf3), but after bumping the MTU from 1500 to 9000 I'm now getting 1.4 Gb/s over the VPN.

derekyang · January 6

Nice tutorial, thanks for sharing !

hotsnow · January 6

Thanks for sharing

ravenchad · January 6

@Daniel15 said:
If you're using HostHatch, make sure you enable "jumbo frames" by setting the MTU to 9000, for best performance. On Debian, edit /etc/network/interfaces and add mtu 9000 under the relevant section (I think both their public and internal networks support jumbo frames).

You can double-check the MTU values by checking /sys/class/net/<device>/mtu:
[email protected]:~#  cat /sys/class/net/eth1/mtu
9000
[email protected]:~#  cat /sys/class/net/wg0/mtu
8920
(as per https://lists.zx2c4.com/pipermail/wireguard/2017-December/002201.html, WireGuard has overhead of 80 bytes per packet, hence the slightly lower MTU for the WireGuard interface).

I was getting 380 Mb/s using WireGuard between two VPSes (measured using iperf3), but after bumping the MTU from 1500 to 9000 I'm now getting 1.4 Gb/s over the VPN.

Both vps are Host Hatch?

how to know if other providers aside Hosthatch supports this?

Daniel15 · January 6

@ravenchad said:
@Daniel15 said:
If you're using HostHatch, make sure you enable "jumbo frames" by setting the MTU to 9000, for best performance. On Debian, edit /etc/network/interfaces and add mtu 9000 under the relevant section (I think both their public and internal networks support jumbo frames).

You can double-check the MTU values by checking /sys/class/net/<device>/mtu:
[email protected]:~#  cat /sys/class/net/eth1/mtu
9000
[email protected]:~#  cat /sys/class/net/wg0/mtu
8920
(as per https://lists.zx2c4.com/pipermail/wireguard/2017-December/002201.html, WireGuard has overhead of 80 bytes per packet, hence the slightly lower MTU for the WireGuard interface).

I was getting 380 Mb/s using WireGuard between two VPSes (measured using iperf3), but after bumping the MTU from 1500 to 9000 I'm now getting 1.4 Gb/s over the VPN.
Both vps are Host Hatch?

how to know if other providers aside Hosthatch supports this?

Both are with HostHatch, yes. I was testing via the internal network (even though it's internal, it's not isolated and is shared between other hosthatch customers at the same location, so encryption is still beneficial).

You'd need to ask the provider whether they support "jumbo frames". If they don't support jumbo frames and you increase the MTU, it'll actually hurt performance quite a bit, as each frame will need to be fragmented (or rejected altogether)

Chocoweb · January 7

Another tip: Create a drop-in service configuration file /etc/systemd/system/nfs-server.service.d/after-wireguard.conf

[Unit]
[email protected]

to make nfs server start after WireGuard is up. This can reduce chance of errors/problems during server reboot

Pixels · January 7

Thank you for the tutorial. I will use this now on my flimsy HostSolutions 2TB server.

icry · January 8

I always end up with a Read Only file system when I try NFS via Wireguard tutorials 😑

Makes me lose my shit 🤐

sumeethannurkar · January 8

Using AUTH_SYS and Export Controls can securely share storage

Erisa · January 8

@icry said:
I always end up with a Read Only file system when I try NFS via Wireguard tutorials 😑

Makes me lose my shit 🤐

Make sure you have rw in the export and that the clients user id (id command) is one that has access the exported folder.

Daniel15 · January 8

@Chocoweb said:
Another tip: Create a drop-in service configuration file /etc/systemd/system/nfs-server.service.d/after-wireguard.conf
[Unit]
[email protected]
to make nfs server start after WireGuard is up. This can reduce chance of errors/problems during server reboot

This is a great tip! Thank you

@sumeethannurkar said: Using AUTH_SYS and Export Controls can securely share storage

I think AUTH_SYS is what's used by default. The connection is unencrypted so you'll want to use encryption (eg. WireGuard) for it to be secure, unless you're on an isolated private network. Not sure which provider isolate their private network... BuyVM definitely do.

@icry said:
I always end up with a Read Only file system when I try NFS via Wireguard tutorials 😑

Makes me lose my shit 🤐

What error are you getting? You likely need to chown the directory to the correct uid, or use idmap to map it across to the right ID.

ravenchad · January 8

@Daniel15 said:
@Chocoweb said:
Another tip: Create a drop-in service configuration file /etc/systemd/system/nfs-server.service.d/after-wireguard.conf
[Unit]
[email protected]
to make nfs server start after WireGuard is up. This can reduce chance of errors/problems during server reboot
This is a great tip! Thank you

@sumeethannurkar said: Using AUTH_SYS and Export Controls can securely share storage

I think AUTH_SYS is what's used by default. The connection is unencrypted so you'll want to use encryption (eg. WireGuard) for it to be secure, unless you're on an isolated private network. Not sure which provider isolate their private network... BuyVM definitely do.

@icry said:
I always end up with a Read Only file system when I try NFS via Wireguard tutorials 😑

Makes me lose my shit 🤐

What error are you getting? You likely need to chown the directory to the correct uid, or use idmap to map it across to the right ID.

I think i got same write permission issue as well with ubuntu at the client side. so i changed folder to chmod -r 777 and it solved my issue. Kindly let me know if there's a better solution to this. This are the small things that are missing in the guide

skorous · January 8

@Chocoweb said: to make nfs server start after WireGuard is up. This can reduce chance of errors/problems during server reboot

It may be worth looking at a Requires= line as well if you don't want it to come up if that interface doesn't.

txtag1 · January 9

Tnx

lyb9b · January 11

@Daniel15 said:
@youandri For backups I use Borgbackup and [Backupninja
command="borg serve --append-only --restrict-to-path /backup/syd01",restrict ssh-ed25519 AAAA........ [email protected]
```

You should mention that in --append-only mode you cannot prune backups in Borg.

You sending prune command from another server as cronjob or sth like this?

Hope that Borg will work as server > client mode in future OOTB

Daniel15 · January 11

@lyb9b said: You should mention that in --append-only mode you cannot prune backups in Borg.

That's a good point. I use --append-only mode so that if someone gains access to any of the backed up servers, they can't just delete all the backups

@lyb9b said: You sending prune command from another server as cronjob or sth like this?

I'm currently not pruning at all. My backups are going to a storage VPS with 2TB space, and less than 400 GB space is used at the moment. I'll figure out the best way to prune them when I get closer to running out of space

One approach is to have two SSH keys: One key for regular backups in --append-only mode, and one key for management that does not run in --append-only mode and thus gets full access (which I'd keep more secure). This is what is documented in Borg's docs: https://borgbackup.readthedocs.io/en/stable/usage/notes.html#append-only-mode.

sumeethannurkar · January 13

Using AUTH_SYS and Export Controls can securely share storage

Howdy, Stranger!

Categories

In this Discussion

How To: Securely Share Storage using NFS

Comments

Howdy, Stranger!

Quick Links

Categories

In this Discussion

How To: Securely Share Storage using NFS

Comments