Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Advertise on LowEndTalk.com
Set up your own truly secure, encrypted and shared file synchronization, aka Dropbox clone
New on LowEndTalk? Please read our 'Community Rules' by clicking on it in the right menu!

Set up your own truly secure, encrypted and shared file synchronization, aka Dropbox clone

TL;DR

This article describes my truly secure, encrypted file synchronization service. It used EncFS and dvcs-autosync which lets me share only the encrypted data and mount that locally to get the plaintext. It works on OS X, Linux and ARM linux. This article has setup instructions for all those platforms.

Diagram

Diagram
Overview of the solution we're building.

My data is in an EncFS encrypted folder. The unencrypted contents are available after unlocking the folder. The encrypted files are synced to an ssh server an to a few other machines and devices using dvcs-autosync. The enryption happens on my machines before the data leaves the to internet.

Preface

Recently I've had to stop using SpiderOak for my file backup and syncronization across machines. The main reason being that there is no ARM version of SpiderOak and the RAM usage was getting out of hand for me. And there still is no open source client, sadly. However, my time with SpiderOak was good, I've paid for it and most of the time it just works fine.

But since I recently bought an ARM Laptop on which I also need my files, it became time to switch to another secure shared file storage. I have a few demands for such a service:

  • It should support synchronization to multiple (more than 2) devices.
  • It has to run on both OS X and any reasonable recent Linux version.
  • It should encrypt files on my machine(s) before going to the internet.
  • It has to be easy to add or remove storage nodes (like vps servers).
  • It has to be open source.
  • It should run on both x86 and ARM (debian armhf) (Chromebook ARM, Raspberry Pi).

Then all current commercial services drop off, including SpiderOak, Bittorrent Sync and git-annex. This resulted in a clever combination of EncFS and dvcs-autosync. Because, in this day and age, you cannot trust any "cloud" provider with your unencrypted data. (And you can only trust those who say they do it securly when they release there source code, wink wink Wuala/Spideroak).

Overview

I'll describe the steps and requirements needed to set this up first. Then we get started with the setup. First we'll set up the server. Then the first Linux client. If needed, steps are provided for adding other Linux clients. Then instructions for OS X are provided. It is a little long, but if you want privacy and security a one time investment is required.

Requirements

Not mandatory:

  • OS X machine (iMac, Macbook) with python 2.6+ (Included in Lion and above), git, xcode, command line tools for xcode and homebrew.

Steps

  • Prepare the SSH/git server

  • Prepare the Linux client

    • Install EncFS
    • Creating the secure EncFS folder
    • Install dvcs-autosync
    • Create an XMPP account
    • Set up dvcs-autosync
    • Special steps for an ARM Chromebook
  • Set up another Linux client

  • Prepare the OS X client

    • Install MacFUSE
    • Install EncFS
    • Get the secure folder
    • Install dvcs-autosync
    • Set up dvcs-autosync

So, lets get started. In about half an hour you have your own secure encrypted file synchronization service.

Set up the SSH server

As said, you'll need an SSH server which will act as your central data repository. Here your encrypted data will reside, and clients push and pull changes to and from here. If you have a few laptops which are not on all the time, this server makes sure all the clients have the most recent data.

If you don't have a VPS, InceptionHosting has good VPS servers for a nice price. (Affiliation link).

I won't cover the installation and setup of the server. SSH, a user account and a passwordless SSH key is all you need. Google can help you with the setup of that.

First install git:

apt-get install git

Now, go to your home folder and create the "repository":

cd ~
git init --bare autosync.git

That's it. Now we are going to set up the clients.

Please read the rest of the tutorial over at Raymii.org!

Thanked by 1Infinity
Quis custodiet ipsos custodes?
https://raymii.org - https://cipherli.st

Comments

  • Just make sure to move the EncFS's dotfile from the folders you will be syncing to the services, that file makes them able to crack your files. (Without having that file cracking the EncFS files will be nearly impossible.)

  • @YellowSloth said:
    Just make sure to move the EncFS's dotfile from the folders you will be syncing to the services, that file makes them able to crack your files. (Without having that file cracking the EncFS files will be nearly impossible.)

    Thanks, added it! Forgot to add it, most of the tutorial is written out of my head. The system is running stable for three weeks now here.

    Quis custodiet ipsos custodes?
    https://raymii.org - https://cipherli.st
  • Thank you for this. I have been wanting to find my own Dropbox alternative. Have tried using Bittorrent Sync, but don't like the idea of someone being able to type in my key and instantly being able to see my files.

    Is there a possibility that this solution could be implemented on Windows as well?

  • RaymiiRaymii Member
    edited September 2013

    @C_Adam said:
    Thank you for this. I have been wanting to find my own Dropbox alternative. Have tried using Bittorrent Sync, but don't like the idea of someone being able to type in my key and instantly being able to see my files.

    Is there a possibility that this solution could be implemented on Windows as well?

    Well, EncFS should be able to run on Windows: http://superuser.com/questions/179150/is-anyone-working-on-an-encfs-client-for-windows, and so does Python and git. It would be a hassle to get working tough. Otherwise via cygwin, or maybe a small Ubuntu virtual machine running the sync setup, and the encfs folder shared to Windows?


    In other news, holy shit frontpage of hackernews:

    hn

    That are only the referal links for the past hour. And I got asked for two podcasts...

    Quis custodiet ipsos custodes?
    https://raymii.org - https://cipherli.st
  • This looks great.

    Can I ask a few questions...

    • I take it there is only one git origin, so if I wanted to use a second VPS for redundancy, it would be a client to it. What happens when the first (origin) goes down?

    • I also take it that as the file is encrypted locally, it is not possible to do delta syncs (ala rsync. EG, if I change two words in a 25Mb document, the entire document will need to be retransmitted).

    • What if I have ~100Gb of documents I wanted synced across devices, but some of those only have a small amount of storage (phone, ssd, etc). How hard is it to only pull certain files on those devices? Say I wanted a handful of text files from various directories stored on my phone, but EVERYTHING stored on my NAS at home?

    Thanks for the write up!

    Checkout my blog. I review VPS providers and mess around with Debian and FreeBSD!

  • @BluBoy said:
    This looks great.

    Can I ask a few questions...

    • I take it there is only one git origin, so if I wanted to use a second VPS for redundancy, it would be a client to it. What happens when the first (origin) goes down?

    You can add both more clients (as in, a vps becomes a dvcs-autosync client), or you can set up an rsync/duplicity backup (nightly sync). Git also has the option to push to multiple masters in one go, but that doesn't work well with dvcs-autosync.

    • I also take it that as the file is encrypted locally, it is not possible to do delta syncs (ala rsync. EG, if I change two words in a 25Mb document, the entire document will need to be retransmitted).

    That is a very big document then. I think you can tweak this a bit with the options of EncFS, but with the recommended paranoid setup every change in a file causes it to be retransmitted yes. However, git is quite intelligent in only uploading the changed stuff, so if you set up less secure EncFS settings less changes to the file and less has to be transmitted. This explains a lot of those options: http://movingtofreedom.org/2007/02/21/howto-encfs-encrypted-file-system-in-ubuntu-and-fedora-gnu-linux/

    • What if I have ~100Gb of documents I wanted synced across devices, but some of those only have a small amount of storage (phone, ssd, etc). How hard is it to only pull certain files on those devices? Say I wanted a handful of text files from various directories stored on my phone, but EVERYTHING stored on my NAS at home?

    Because everything is encrypted, and you only sync the encrypted blobs you don't know what is what until you encrypt it. There (luckally) is no easy way to match those two together, so just sync certain files is not possible. What you can do, is set up multiple repositories, one for the phone and one with all the data.

    git-annex is very good for the use case you just described: http://git-annex.branchable.com/

    Thanks for the write up!

    Quis custodiet ipsos custodes?
    https://raymii.org - https://cipherli.st
Sign In or Register to comment.