Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


How To: Securely Share Storage using NFS
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

How To: Securely Share Storage using NFS

I just wrote a basic guide on how to use WireGuard, NFS and optionally cachefilesd for local caching of NFS files, but this site keeps giving me a broken Cloudflare CAPTCHA when I try to post it:

So I posted it to my own blog instead: https://d.sb/2020/12/nfs-howto

Seeing as a bunch of people here have probably gotten storage VPSes during Black Friday and Christmas sales and are thinking of ways to use the storage space, I thought you'd find it useful :)

«1

Comments

  • Good tutorial.

    Do you have how to backup vps to storage vps with easy?

  • Thanks @Daniel15

  • Daniel15Daniel15 Veteran
    edited January 2021

    @youandri For backups I use Borgbackup and Backupninja. Both are available in Debian (apt install borgbackup backupninja). I haven't written a tutorial for it, but here's some very rough unedited notes from a Google Doc I use when setting up my own servers:

    On backup server (storage server that holds the backups), add user and create backup dir:

    adduser backup-syd01 --disabled-password
    mkdir /backup/syd01
    chown backup-syd01:backup-syd01 /backup/syd01
    chmod 0700 /backup/syd01
    

    (where syd01 is the name of the origin server)

    On origin (server you want to back up), generate SSH key as root. Do NOT set a passphrase:

    ssh-keygen -o -a 100 -t ed25519 -f ~/.ssh/id_ed25519 -C "[email protected]"
    

    Get the public key from ~/.ssh/id_ed25519.pub. On backup server, create authorized_keys for user using this public key, and restrict it to only allow Borgbackup (no regular SSH access using this key):

    command="borg serve --append-only --restrict-to-path /backup/syd01",restrict ssh-ed25519 AAAA........ [email protected]
    

    On origin, create Borg repo and backup key:

    borg init --encryption=keyfile-blake2 [email protected]:/backup/syd01
    borg key export [email protected]:/backup/syd01 key
    cat key
    

    Enter a randomly generated password when prompted. Store key AND the password in a safe place (eg in LastPass) as you need BOTH to decrypt the backups.

    Then you can configure Backupninja using ninjahelper, and modify / etc/backupninja.conf to set when the backups run, where it sends emails, and if it sends emails on success or only on failure.

  • Nice write-up! I'm curious if this is resilient against a sometimes shitty network?

  • Daniel15Daniel15 Veteran
    edited January 2021

    @its420somewhere said:
    Nice write-up! I'm curious if this is resilient against a sometimes shitty network?

    By default it'll keep retrying indefinitely, so in theory it should handle unreliable networks. If a client tries to read from or write to the NFS share, and the NFS server is down, it'll block until the server is back up (or you give up and kill the process :tongue: ).

    You can change it to stop retrying after a period of time (the soft option) but that can result in data loss. For example, if a write times out, that write will be lost. I would recommend always using hard mode, which is the default. You can see the nfs(5) man page for more details :)

    Thanked by 1its420somewhere
  • What is the advantage of using NFS over mounting SFTP as a network drive or using SMB with user authentication?

    Thanked by 10xbkt
  • @Daniel15 said:

    @its420somewhere said:
    Nice write-up! I'm curious if this is resilient against a sometimes shitty network?

    By default it'll keep retrying indefinitely, so in theory it should handle unreliable networks. If a client tries to read from or write to the NFS share, and the NFS server is down, it'll block until the server is back up (or you give up and kill the process :tongue: ).

    You can change it to stop retrying after a period of time (the soft option) but that can result in data loss. For example, if a write times out, that write will be lost. I would recommend always using hard mode, which is the default. You can see the nfs(5) man page for more details :)

    Awesome! Thanks for clarifying that, even though my question was quite vague you answered the concerns I had about dropouts etc.

    I've been using rclone and sftp and the dropouts really take the jam out of my donut.

  • SP7SP7 Member

    Thanks for the great write-up! How long does it keep the files in cache? Does it get removed after some time?

  • @SP7 said:
    Thanks for the great write-up! How long does it keep the files in cache? Does it get removed after some time?

    The cache keeps growing until the free disk space or inodes drops below a particular limit, at which point it'll start removing the least recently used files from the cache. To change the max cache size, you can modify the cachefilesd config, or use a separate partition just for the cache.

    Thanked by 1SP7
  • SP7SP7 Member
    edited January 2021

    Edit: Nevermind :sweat_smile:

  • Thanks @Daniel15 for the tutorial.

    I was wondering what are the other options that does a similar thing? this way is the fastest way?

  • Daniel15Daniel15 Veteran
    edited January 2021

    @Kassem said: I was wondering what are the other options that does a similar thing?

    There's a few other options:

    SSHFS is one of the most popular. It just uses SSH, so no extra configuration is required on the server side, as long as SFTP is enabled on your server (it's enabled by default). It's fairly stable but can have reliability issues if you try to read/write a lot of files in parallel (SFTP really isn't designed for that use case). SSHFS does not have any active contributors so any bugs you encounter are likely to not get fixed unless you fix them yourself :)

    SMB is popular too. It's the protocol Windows uses for file sharing. You have to install "Samba" to use it. It does support username and password authentication out-of-the-box. I think SMB3 supports encryption but I'm not 100% sure. I haven't touched Samba for many many years now so I'm not sure how it compares to NFS these days. It used to have a reputation for being insecure back in the Windows 98 days, but newer versions are better.

    SSHFS and Samba are both apps you need to install and run, whereas NFS is built in to the Linux kernel (the bits you install are just the commands to administer it). In theory, kernel code is faster and more efficient than "userland" (non-kernel) code.

    Here's an article I just found with a performance comparison: https://blog.ja-ke.tech/2019/08/27/nas-performance-sshfs-nfs-smb.html

  • Thanks for the details @Daniel15

    What about Gluster over Wireguard without replication? Would that be comparable to NFS over Wireguard at all?

    I see this: https://www.howtoforge.com/creating-an-nfs-like-standalone-storage-server-with-glusterfs-3.2.x-on-ubuntu-12.10

    (outdated and doesn't secure it with Wireguard though)

  • wouldn't using wireguard also route all your internet traffic through the tunnel? making this not so efficient?

  • Thank you for explaining the cache part. The rest I had already mastered on my own. Runs great so far.

  • Daniel15Daniel15 Veteran
    edited January 2021

    @jugganuts said:
    wouldn't using wireguard also route all your internet traffic through the tunnel? making this not so efficient?

    No. In the tutorial I only use a single IP in the AllowedIPs in each WireGuard peer (eg. 10.123.0.1/32), meaning just that one IP is routed to that node via the VPN. This is usually called a "peer-to-peer" or "point-to-point" VPN. This is the way that WireGuard works by default out-of-the-box. No packet forwarding is taking place and it's just peers directly communicating with other peers.

    To route all your traffic through WireGuard (which is often what people think of when they hear "VPN"), you'd need to allow all IPs (0.0.0.0/0), and also add some iptables rules to do the routing. I don't do that.

    @Kassem said: What about Gluster over Wireguard without replication? Would that be comparable to NFS over Wireguard at all?

    I haven't tried Gluster so I'm not too sure about that.

    Thanked by 1its420somewhere
  • ravenchadravenchad Member
    edited January 2021

    @Daniel15 said:
    I just wrote a basic guide on how to use WireGuard, NFS and optionally cachefilesd for local caching of NFS files, but this site keeps giving me a broken Cloudflare CAPTCHA when I try to post it:

    So I posted it to my own blog instead: https://d.sb/2020/12/nfs-howto

    Seeing as a bunch of people here have probably gotten storage VPSes during Black Friday and Christmas sales and are thinking of ways to use the storage space, I thought you'd find it useful :)

    thanks @Daniel15 , for a good tutorial,. sorry for noob question. after establishing the connection with WireGuard, how could I test the up/down speed between the server and client. is it just by downloading files or there are other ways? thanks

  • ErisaErisa Member
    edited January 2021

    @ravenchad said: thanks @Daniel15 , for a good tutorial,. sorry for noob question. after establishing the connection with WireGuard, how could I test the up/down speed between the server and client? thanks

    You'd want to use a program like iperf3.
    On a basic level (Quick speed test without tuning options) this would just be iperf3 -s -B 10.123.0.1 on one end and iperf3 -c 10.123.0.1 on the other. Of course replace the IP; -s is the server end and -c is the client. I took the IP from the tutorial in question here.

    Thanked by 1Daniel15
  • @Erisa said:

    @ravenchad said: thanks @Daniel15 , for a good tutorial,. sorry for noob question. after establishing the connection with WireGuard, how could I test the up/down speed between the server and client? thanks

    You'd want to use a program like iperf3.
    On a basic level (Quick speed test without tuning options) this would just be iperf3 -s -B 10.123.0.1 on one end and iperf3 -c 10.123.0.1 on the other. Of course replace the IP; -s is the server end and -c is the client. I took the IP from the tutorial in question here.

    great. thanks

  • @Daniel15 in your experience have you found wireguard to be good at reconnecting if the connection drops etc?

  • @its420somewhere said:
    @Daniel15 in your experience have you found wireguard to be good at reconnecting if the connection drops etc?

    You'll be glad to hear aside from occasional handshakes, WireGuard does not really have much of a concept of "reconnecting". When you send a packet through a created WireGuard interface it is simply encrypted with the private key and sent to its appropriate endpoint regardless of current state. At a simplified level this is all that WireGuard is doing, encrypting packets over a virtual network. You define who you can send packets to and who can send you packets, over which ports and using which encryption keys.

    In its bare form it provides unparalleled performance across shaky networks, functioning the same as any other (UDP) internet protocol traffic would. Using WireGuard on a mobile device with a mobile network feels like it doesn't even have it active, connection is as shaky as your location and supporting applications.
    So the handling of a situation where your network is dropping frequently would be down to the other protocol in use, for this example it would be NFS but in other cases could be SSH, HTTP, etc.

    You can configure it with a persistent keepalive handshake, but that's mostly to avoid confusing some NATs/firewalls and doesn't do much except keep the connection appearing alive even when its idling (So the NAT doesnt assume you're done with that endpoint and cut you off).

    (Feel free to correct me if I'm wrong, this is just what I've come to learn over reading and use)

  • Daniel15Daniel15 Veteran
    edited January 2021

    @ravenchad said: after establishing the connection with WireGuard, how could I test the up/down speed between the server and client.

    I'd use iperf3 like @Erisa said. Note that it won't be as fast as an unencrypted connection, as encryption isn't free (it needs some CPU power). However, WireGuard should be faster than some other VPN systems like OpenVPN.

    @its420somewhere said: in your experience have you found wireguard to be good at reconnecting if the connection drops etc?

    WireGuard doesn't really 'connect' nor resend lost packets nor handle reordering packets that arrive out of order, as it's UDP. UDP just sends packets over the internet without checking if they arrive properly or in the right order.

    Stuff like that is at a higher layer than WireGuard. When you perform TCP connections over the VPN (like HTTP requests, NFS connections, etc), those connections handle the reconnections and retransmissions, as part of TCP. WireGuard doesn't have to do it, as the protocols being tunneled over the VPN already do it. So, basically what @Erisa said.

    This is also why it's usually a bad idea to use a TCP connection for a VPN. It's sometimes useful to bypass proxies (like at school or a workplace that blocks UDP traffic), but there's much more overhead, as you're tunnelling TCP over TCP. The nature of TCP means that all the packets must arrive in exactly the same order as they're sent, with nothing dropped, which makes it a lot slower.

  • Wiregurd uses less memory and CPU than other protocols VPN like openvpn/pptp/l2tp

  • @Daniel15 said:

    @ravenchad said: after establishing the connection with WireGuard, how could I test the up/down speed between the server and client.

    I'd use iperf3 like @Erisa said. Note that it won't be as fast as an unencrypted connection, as encryption isn't free (it needs some CPU power). However, WireGuard should be faster than some other VPN systems like OpenVPN.

    @its420somewhere said: in your experience have you found wireguard to be good at reconnecting if the connection drops etc?

    WireGuard doesn't really 'connect' nor resend lost packets nor handle reordering packets that arrive out of order, as it's UDP. UDP just sends packets over the internet without checking if they arrive properly or in the right order.

    Stuff like that is at a higher layer than WireGuard. When you perform TCP connections over the VPN (like HTTP requests, NFS connections, etc), those connections handle the reconnections and retransmissions, as part of TCP. WireGuard doesn't have to do it, as the protocols being tunneled over the VPN already do it. So, basically what @Erisa said.

    This is also why it's usually a bad idea to use a TCP connection for a VPN. It's sometimes useful to bypass proxies (like at school or a workplace that blocks UDP traffic), but there's much more overhead, as you're tunnelling TCP over TCP. The nature of TCP means that all the packets must arrive in exactly the same order as they're sent, with nothing dropped, which makes it a lot slower.

    thanks @Daniel15 , by the way. anyone having issue with etc/fstab config. it seems not working with ubuntu18. i tried it multiple times and its not attaching after reboot. however, using manual "mount -a" it is working.

  • edited January 2021

    @ravenchad said:

    @Daniel15 said:

    @ravenchad said: after establishing the connection with WireGuard, how could I test the up/down speed between the server and client.

    I'd use iperf3 like @Erisa said. Note that it won't be as fast as an unencrypted connection, as encryption isn't free (it needs some CPU power). However, WireGuard should be faster than some other VPN systems like OpenVPN.

    @its420somewhere said: in your experience have you found wireguard to be good at reconnecting if the connection drops etc?

    WireGuard doesn't really 'connect' nor resend lost packets nor handle reordering packets that arrive out of order, as it's UDP. UDP just sends packets over the internet without checking if they arrive properly or in the right order.

    Stuff like that is at a higher layer than WireGuard. When you perform TCP connections over the VPN (like HTTP requests, NFS connections, etc), those connections handle the reconnections and retransmissions, as part of TCP. WireGuard doesn't have to do it, as the protocols being tunneled over the VPN already do it. So, basically what @Erisa said.

    This is also why it's usually a bad idea to use a TCP connection for a VPN. It's sometimes useful to bypass proxies (like at school or a workplace that blocks UDP traffic), but there's much more overhead, as you're tunnelling TCP over TCP. The nature of TCP means that all the packets must arrive in exactly the same order as they're sent, with nothing dropped, which makes it a lot slower.

    thanks @Daniel15 , by the way. anyone having issue with etc/fstab config. it seems not working with ubuntu18. i tried it multiple times and its not attaching after reboot. however, using manual "mount -a" it is working.

    Maybe the data you entered not complete, the format should be like this:

    # <file system>     <dir>       <type>   <options>   <dump> <pass>
    10.10.0.10:/backups /your/local/mount nfs      defaults    0       0
    
  • ravenchadravenchad Member
    edited January 2021

    @chocolateshirt said:

    @ravenchad said:

    @Daniel15 said:

    @ravenchad said: after establishing the connection with WireGuard, how could I test the up/down speed between the server and client.

    I'd use iperf3 like @Erisa said. Note that it won't be as fast as an unencrypted connection, as encryption isn't free (it needs some CPU power). However, WireGuard should be faster than some other VPN systems like OpenVPN.

    @its420somewhere said: in your experience have you found wireguard to be good at reconnecting if the connection drops etc?

    WireGuard doesn't really 'connect' nor resend lost packets nor handle reordering packets that arrive out of order, as it's UDP. UDP just sends packets over the internet without checking if they arrive properly or in the right order.

    Stuff like that is at a higher layer than WireGuard. When you perform TCP connections over the VPN (like HTTP requests, NFS connections, etc), those connections handle the reconnections and retransmissions, as part of TCP. WireGuard doesn't have to do it, as the protocols being tunneled over the VPN already do it. So, basically what @Erisa said.

    This is also why it's usually a bad idea to use a TCP connection for a VPN. It's sometimes useful to bypass proxies (like at school or a workplace that blocks UDP traffic), but there's much more overhead, as you're tunnelling TCP over TCP. The nature of TCP means that all the packets must arrive in exactly the same order as they're sent, with nothing dropped, which makes it a lot slower.

    thanks @Daniel15 , by the way. anyone having issue with etc/fstab config. it seems not working with ubuntu18. i tried it multiple times and its not attaching after reboot. however, using manual "mount -a" it is working.

    Maybe the data you entered not complete, the format should be like this:

    # <file system>     <dir>       <type>   <options>   <dump>   <pass>
    10.10.0.10:/backups /your/local/mount nfs      defaults    0       0
    

    thanks..i will try.. but i am using the same config as mentioned in @Daniel15 blog. like below

    To automatically mount the directory on boot, modify /etc/fstab:

    10.123.0.1:/data/storage /mnt/data nfs4 auto,vers=4.2

  • Here's the exact line I use in /etc/fstab on one of my systems, and it works fine:

    chi01.vpn.d.sb:/data/nfs/chi03 /mnt/data nfs4 auto,fsc,vers=4.2,async
    

    @ravenchad said: it seems not working with ubuntu18. i tried it multiple times and its not attaching after reboot. however, using manual "mount -a" it is working.

    That's strange... I think booting is supposed to do the same thing as mount -a. I'm not sure what the problem could be here.

  • Daniel15Daniel15 Veteran
    edited January 2021

    If you're using HostHatch, make sure you enable "jumbo frames" by setting the MTU to 9000, for best performance. On Debian, edit /etc/network/interfaces and add mtu 9000 under the relevant section (I think both their public and internal networks support jumbo frames).

    You can double-check the MTU values by checking /sys/class/net/<device>/mtu:

    root@chi03:~#  cat /sys/class/net/eth1/mtu
    9000
    root@chi03:~#  cat /sys/class/net/wg0/mtu
    8920
    

    (as per https://lists.zx2c4.com/pipermail/wireguard/2017-December/002201.html, WireGuard has overhead of 80 bytes per packet, hence the slightly lower MTU for the WireGuard interface).

    I was getting 380 Mb/s using WireGuard between two VPSes (measured using iperf3), but after bumping the MTU from 1500 to 9000 I'm now getting 1.4 Gb/s over the VPN.

  • Nice tutorial, thanks for sharing !

  • Thanks for sharing ;)

Sign In or Register to comment.