Postfix/Dovecot distributed setup

perennate

I've been playing around with Dovecot's dsync replication feature to get a degree of redundancy in a low-volume e-mail system. The basic setup is:

• Two incoming mail servers
• Two main mail servers for IMAP / user SMTP

The separation of incoming mail servers isn't too important and won't really be covered in this guide.

This guide doesn't add anything new, but it merges some tutorials and shows how they can work together to get a nice setup.

Goals

We don't want the failure of a single mail server to disrupt our system. However, the goal here is not to get a consistency guarantee that any accepted e-mail will be available even if a server goes down; there is still a chance that some e-mails will be lost until the server that received it comes back online. Instead, we want these properties in the face of single-server failures:

• New e-mails can be received and user can see them
• Most existing e-mails can be retrieved
• New e-mails can be sent

If you want a stronger consistency guarantee, something like putting your Maildir (or whatever mail storage) on GlusterFS or another distributed filesystem may work for you; however, this introduces latency that we don't want with a small mail server. So instead, we'll use dsync, which replicates the mail asynchronously (in the background).

We will use DNS round-robin system for the two main mail servers so that, if one of them is down, clients will eventually connect to the other one.

dsync might not work with a high-volume mail server, I'm not sure about the performance.

MySQL setup

We will be using virtual domains and email addresses. A master-slave-replicated MySQL database will be needed to store this data so that the configuration is retrievable by Dovecot when one of the participating servers fail. Dovecot only needs read access to the database, so there's no issue when the master fails.

This is already covered in other guides:

• https://www.digitalocean.com/community/tutorials/how-to-set-up-master-slave-replication-in-mysql
• http://lowendbox.com/blog/getting-started-with-mysql-master-slave-replication/
• For SSL: https://dev.mysql.com/doc/refman/5.1/en/replication-solutions-ssl.html

SSL isn't critical since the database will only store the basic configuration information, but it's not hard to setup so you might as well.

Postfix/Dovecot initial setup

Here we'll setup the main mail servers. First, exchange SSH keys between the servers so that you can easily copy configuration files between them (or you can use your favorite orchestration framework).

Now, follow this guide to complete initial setup: https://www.linode.com/docs/email/postfix/email-with-postfix-dovecot-and-mysql

Obviously when you're following the guide, you'll want to make sure the MySQL database that you're using is the one that's replicated between the two servers.

Make sure that the hostnames configured in Postfix on the two servers are different, otherwise there may be some problems later. So you can do mail1.example.com and mail2.example.com. However, we (may) want both of the servers to be accessible by clients from the same DNS hostname. So do use the same SSL certificate and external hostname.

Also configure DNS however you like, something similar to this probably:

• MX record mail.example.com
• DNS round robin: mail A 1.1.1.1 and mail A 1.1.1.2
• Then mail1 A 1.1.1.1 and mail2 A 1.1.1.2

Setup dsync

Now that you have two mostly identical mail servers with replicated MySQL, we can add dsync to synchronize the e-mails. dsync is a Dovecot feature which the "replication" plugin uses to support the setup that we want.

There's a small issue if you followed the Linode guide, where we cannot list the users. The replication feature will need to be able to list all user accounts so that it knows which ones it should replicate. We can do a few small modifications to the Dovecot configuration so that this is supported.

• conf.d/auth-sql.conf.ext:

  • change driver=static; args=...
  • to driver=sql; args=/etc/dovecot/dovecot-sql.conf.ext

• dovecot-sql.conf.ext: since we've updated driver to use this file, we'll need to add queries for the user DB. There are two queries we want to include:

  • user_query = SELECT 'vmail' AS uid, 'vmail' AS gid, '/var/mail/vhosts/%d/%n' AS home;
  • iterate_query = SELECT email AS user FROM virtual_users

Now we're all set to follow the instructions in the Dovecot replication guide. You only need to follow the top section of that guide, anything under "dsync over TCP connections" is discussing alternatives (which you can look at if desired, but the default SSH keys works well). Note that you'll have to create SSH keys for the vmail user and exchange them between mail1/mail2 so that vmail@mail1 and vmail@mail2 can connect to each other.

Try restarting Postfix/Dovecot now and check for errors. You should be able to connect via IMAP / SMTP if everything worked.

Issues with single user accessing different servers

The Dovecot replication page above mentions that there may be annoyances if a user is being directed to different Dovecot servers on different requests (user might re-download mail). We can solve these problems by using Dovecot director service, but this is a bit complicated. Alternatively, we can create another DNS record "mail-online.example.com" and add some DNS failover solution to switch it to the other mail server when one of them goes offline (so maybe mail-online.example.com points to mail1.example.com usually, but swaps to mail2.example.com if mail1 fails). We can also require users to manually failover if server failures are rare enough.

Services that need to be on a single server

Some e-mail related services need to be on a singe server. For example, mailman is a popular mailing list system without support for a distributed setup. This is problematic since each of our mail servers only sees a portion of the overall mail traffic; we have only synchronized the e-mail that ends up at our users. To handle these other cases, you'll want to put them on a separate additional server, e.g. mailman.example.com. Install Postfix and mailman (or whatever software you want) on this server. Then:

• Create virtual aliases in the MySQL database so that mail for [email protected] gets directed to [email protected] by your main mail servers; Postfix will see this alias and know that it should forward the mail to another server (it'll use the A record of mailman.example.com)

  • Keep in mind that BOTH of our main mail servers will be directing mail to mailman.example.com; so, mailman.example.com will see all the e-mail that it needs to see

• Restrict the Postfix service on mailman.example.com: smtpd_relay_restrictions = permit_mynetworks reject; update mynetworks to include mail1.example.com/mail2.example.com

• Configure Postfix on mailman.example.com to forward outgoing mail back to our main mail servers: relayhost = mail.example.com (if one main mail server goes down, it'll eventually use the other one)

Of course if mailman.example.com goes down, then our mailing lists will be unavailable until it comes back online, but there's not much we can do about that.

Note: some services, possibly including mailman, have a SMTP pull feature that you can use to pull the emails from an IMAP server instead of having the MTA push it to the application. That is a reasonable alternative, although the solution outlined above would reduce latency and provide better performance.

Separate incoming/outgoing mail servers

It's easy to separate out incoming mail servers and outgoing mail servers. For incoming mail servers, create at least two servers for this purpose. Configure any blacklist-checking, spam-filtering, virus-filtering, etc. that you want on both of the servers.

Then, set relayhost = mail.example.com. That should be all you need. The DNS configuration simply involves creating a separate MX entry for each incoming mail server; you can set the priority however you desire.

For outgoing mail servers, we'll need another round-robin DNS record since Postfix doesn't support relayhost load balancing. So, create outgoing1.example.com and outgoing2.example.com, and have outgoing.example.com include both of them. Then set relayhost = outgoing.example.com on both of your main mail servers.

.

Open to any suggestions and better solutions!

jar

perennate said: however, this introduces latency that we don't want with a small mail server

So true, so true... the great latency of MXroute of 2015.

perennate

Jar said: So true, so true... the great latency of MXroute of 2015.

You tried storing mail on DFS? That sounds pretty cool, how did it go / still using it?

jar

@perennate said:
You tried storing mail on DFS? That sounds pretty cool, how did it go / still using it?

I used SSHFS, while obviously not the finest choice I wanted it to work consistently in OVZ, and with about 250 average incoming and about the same outgoing per hour, it was too often slow to read (despite being in same DC over private network, excellent sustained bandwidth), and Dovecot kept randomly writing files that were root owned. I had a cron running to correct permissions, I couldn't accurately monitor the SSHFS mount since failure would not result in any reliably measurable circumstance besides the endless latency of an "ls" command.

I rsync'd everything back over and dropped the plan. Your way sounds better