Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Multicore/Mulithread file encryption on Linux
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Multicore/Mulithread file encryption on Linux

alfinderaualfinderau Member
edited October 2016 in General

Hi guys,

on and off I send large files (> 10 GB) to my backup server allowing only unsecure ftp.

Encrypting by PGP puts you to sleep as it only uses one core. But my server provides 8 of them which I'd be happy to feed with work.

I've done some enquires on Google but somehow I can't find a multicore/multithread app for Debian/Ubuntu.

Are there any out? Or does nobody need them?

Thanks for sharing your experience.
Alf

(N.B. I only want to encrypt single files not a whole disk or partition)

Comments

  • joepie91joepie91 Member, Patron Provider

    Strong encryption usually shouldn't be parallelizable. Almost all modern block cipher modes operate on the results of past "blocks". The only exception I can think of here is GCM, and if I recall correctly, it's tricky to implement securely (especially in cross-platform code).

    In short: if you find something that claims to be able to use multiple threads for encryption, be extremely wary, because it's probably broken.

  • Well, he mentioned "files" so he could launch the encryption of different files simultaneously.

  • joepie91joepie91 Member, Patron Provider

    @UrDN said:
    Well, he mentioned "files" so he could launch the encryption of different files simultaneously.

    If they're encrypted separately, sure. But at that point, why not just use something like parallel?

  • rm_rm_ IPv6 Advocate, Veteran

    Yeah here you go: http://blog.tkassembled.com/412/encrypt-and-decrypt-in-parallel-with-gpg/

    If you only have one 10GB file at a time to encrypt, you could split it into 1GB (or whatever) chunks first, or preferably at the time of creation via a pipe, using the split tool.

  • Thanks to all. Very interesting.

    Indeed, usually I have to encrypt one single file > 10 GB. The split method rm_ mentioned sounds interesting. I'll give it a try.

  • Ok, I tried it. It works, but splitting is also very time-consuming, at least on my harddisk based server.

    I understand what joepie91explained above but isn't it strange that there is no 1-click-and-be-happy solution in 2016's encryption world when we all run machines with umpteen cores?

  • @alfinderau said:
    I understand what joepie91explained above but isn't it strange that there is no 1-click-and-be-happy solution in 2016's encryption world when we all run machines with umpteen cores?

    Not so strange. The Unix philosophy is to do one thing well, and the common mechanism of piping commands is inherently serial. I mean, what would you do even if you did have a version of split that operated in parallel using multiple threads? To avoid writing to disk you'd have to have a mechanism that the shell understands as a way to feed multiple output streams into the "next" command. I guess you could work something up with named pipes, but it's hard to say how well it would work if split isn't written to be multithreaded.

  • rm_rm_ IPv6 Advocate, Veteran
    edited October 2016

    alfinderau said: splitting is also very time-consuming

    That's why I said you could do it at the time of backup creation, via pipe. E.g.:

    tar -cO /home/ | pbzip2 -c1 | split -b 1G - `date +%F`-home.tar.bz2.

    This will write tar+bzipped content directly into 1GB-sized chunks.

    (and install pbzip2 if you don't have it, also good for multi-core CPUs)

    But yeah if you have an HDD server, encrypting 8 files in parallel after that may be slow due to all the constant HDD seeks.

    impossiblystupid said: what would you do even if you did have a version of split that operated in parallel using multiple threads

    That's a solved problem, there are pbzip2, plzip, plzo, pxz, maybe more. That there's no "pGPG" is unfortunate, but can be worked around as shown above.

    Thanked by 1netomx
  • By default, GPG uses "asymmetric" or public key cryptography for file encryption. That is a slow operation itself.
    Instead use hybrid cryptography:
    1. Create a symmetric key
    2. Use that key to encrypt your large file with openssl
    3. Encrypt your key with gpg and your public key
    4. Upload encrypted backup and encrypted key to your backup server
    5. Remove unencrypted key

    See tutorial here: http://www.czeskis.com/random/openssl-encrypt-file.html

    That would be around 3-4 times faster.

  • rm_rm_ IPv6 Advocate, Veteran
    edited October 2016

    yourserverse said: By default, GPG uses "asymmetric" or public key cryptography for file encryption. That is a slow operation itself.

    IIRC it's something along the lines of only the random session key (per file?) being encrypted using the public key crypto, and the actual file is encrypted using just the normal symmetric-like algorithm using that random temporary key. Yes, the public key encryption is slow, in fact it's way too slow to be practical, that's why it's done like this. And not just here, but also in stuff like HTTPS.

    In short, GPG (and HTTPS) already does what you described, automatically under the hood.

    Thanked by 1Abdussamad
  • @rm_ said:

    impossiblystupid said: what would you do even if you did have a version of split that operated in parallel using multiple threads

    That's a solved problem, there are pbzip2, plzip, plzo, pxz, maybe more. That there's no "pGPG" is unfortunate, but can be worked around as shown above.

    I was talking more in the context of general mechanisms for running parallel processes so that the task could be accomplished without the need to rewrite every command or, alternatively, a standard for dealing with multiple I/O streams so that it makes more sense to update more commands to run with threads.

    That said, I did work up a toy script that creates multiple named pipes for split to write to, and it does indeed work as expected, but the main speedup is in avoiding disks writes/reads, because split itself is still operating on the file in a single thread. The best way to exercise the CPU is your suggestion to split at the time of creation, and then running multiple encryption processes on the resulting files in parallel.

  • joepie91joepie91 Member, Patron Provider

    rm_ said: Yes, the public key encryption is slow, in fact it's way too slow to be practical, that's why it's done like this.

    It's not just for performance reasons. It's also horribly insecure to encrypt chunks directly with the public key.

  • xyzxyz Member

    In theory you could alternate blocks, interleaving them into multiple CBC streams. But it's probably just easier and cleaner to use a CTR based block mode for encryption, for parallel processing.

    But does your CPU support AES-NI? If so, make sure your application is using AES128 and supports AES acceleration. AES-NI is usually fast enough on a single thread that you shouldn't need to multi-thread in most cases.

Sign In or Register to comment.