Let's Encrypt and Anycast

Giulio

I've got all the three locations from BuyVM in order to deploy my personal website on the freely provided anycast IP Address both for DNS and HTTP.

I had issues finding a proper way to verify the main domain in all the three location in order to let every node renew its certificate automatically with acme.sh without the need of a master node or any custom script/code.

The requirements:

1. Every node must have a different private key
2. The nodes do not interact with each other
3. Must work with acme.sh out of the box
4. Must renew automatically

Although the solution is simple, it took me a couple of days to think of it so that's the reason why I'm sharing it.

Assume you have three nodes:

lu.mydomain.com
lv.mydomain.com
ny.mydomain.com

Here's a sample nginx config for lu.mydomain.com node with only the relevant parts:

# Default server to redirect to https main domain, when handling non existant *.mydomain.com subdomains
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name _;
    return 301 $scheme://mydomain.com$request_uri;
}

# Server for Let's Encrypt validation
server {
    listen 80;
    listen [::]:80;
    server_name mydomain.com www.mydomain.com lu.mydomain.com;

    location @anycast_redirect {
        return 301 $scheme://lv.mydomain.com$request_uri;
    }

    location /.well-known {
        error_page 404 = @anycast_redirect;
        alias /var/www/mydomain.com/acme/.well-known;
    }

    root /var/www/mydomain.com/htdocs;

    location / {
        return 301 https://mydomain.com$request_uri;
    }
}

# Https server with everything else
server {
    listen 443 ssl http2 default_server;

[...]

In the HTTP-01 Acme specification there's written that the validation server follows up to 10 redirects so this approach can work with a limited number of servers in the anycast pool.
The logic is that each server, if trying to verify a nonexistent Acme challenge will automatically redirect to another one in loop:

lu.mydomain.com -> lv.mydomain.com
lv.mydomain.com -> ny.mydomain.com
ny.mydomain.com -> lu.mydomain.com

With this configuration, each node can now issue and renew with:

acme.sh --ecc --issue/--renew -d mydomain.com -w /var/www/mydomain.com/acme/ --fullchain-file /etc/ssl/certs/mydomain.com.pem --key-file /etc/ssl/private/mydomain.com.key

Thoughts? Is there a simpler solution? This is of course an experiment for an experimental personal website that doesn't even need to be anycast in the first place

aiden1

dns TXT record verification get * wildcard

freerangecloud

Any reason you're using HTTP auth and not DNS-based auth (assuming your DNS server has an API of course)

Daniel15

Huh, this is a pretty clever solution. I would have just gone with DNS verification though

@freerangecloud said: (assuming your DNS server has an API of course)

I'd recommend acme-dns as most DNS server APIs have way too much power in their access tokens (eg. ability to edit any record of any zone). acme-dns reduces your attack surface as it only serves ACME TXT records.

Let's Encrypt is totally fine with IPv6-only DNS servers, so you can just run acme-dns on a spare IPv6 address.

Giulio

@aiden1 said:
dns TXT record verification get * wildcard

Sorry that I didn't mention it, but I prefer single certificates to wildcard.

@freerangecloud said:
Any reason you're using HTTP auth and not DNS-based auth (assuming your DNS server has an API of course)

I'm self hosting DNS on the same machines with nsd. Since in this case one of the anycast nodes is a master and the other two are slaves, updating the master from the slaves would require additional work compared to the HTTP only solution.

@Daniel15 said:
Let's Encrypt is totally fine with IPv6-only DNS servers, so you can just run acme-dns on a spare IPv6 address.

Apart from the fact that it requires an additional software, I didn't think of it and it would ne another clever solution!

SplitIce

As someone who has gone down this track (Anycast + LE) let me say your initial solution is certainly inspired.

Not something anyone should do in a production setting of course. But a unique abuse of technology. I love it.