Dovecot POP3 mails get downloaded again because of changed uidl?

Vita

Hi,

Let me introduce you to the problem. There are multiple mail servers behind a haproxy service. Let's say we have mail01, mail02 and mail03. They all connect to central NFS storage and store the necessary files (index, control,home and maildir files) in users directory (/nfs/users/userX) on the NFS system.

The users are managed through MySQL database that is replicated on all mail nodes.

I've been in the process of separating index, control and home files to from the maildir location. The maildir is still kept the same to /nfs/users/userX. The thing is when the rsync is done and all the files and directories have been migrated to appropriate in this case /nfs/dovecot/index/userX, /nfs/dovecot/control/userX, /nfs/dovecot/home/userX, it seems that somehow and for some reason, dovecot on the new location recreates the dovecot-uidvalidity, or rather to be precise increments this value. It does not do this immediately but rather when a user request emails via POP3.

The dovecot server for POP3 is configured with this:

protocol pop3 {
  disable_plaintext_auth = no
  pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
  pop3_uidl_format = %08Xv%08Xu
}

Meaning that this uidl changes because the dovecot-uidvalidity file has been changed. This then renders in users using POP3 to redownload all mails from the server.

Mails are kept on the server after downloading, because IMAP is used so no deleting is possible.

Since I've done rsync and switched the locations of index, control and home files to point to the new ones (mails itself remain at the old location) what may be the trigger that changes this dovecot-uidvalidity?

Also if someone can explain the formatting to me with this pop3_uidl_format?

I've found some info on https://dovecot.org/list/dovecot/2005-October/009357.html but still, this does not explain the %08, but it seems from the link, and since the dovecot-uidvalidity has been changed to be the case why the mails are downloaded again.

If I switch back to the old paths because the mails have been set as unread and downloaded again local mail clients remember the new uid and then downloads the mails again.

If you need more info, please let me know.

TL; DR; When using POP3 mails get marked as unread and downloaded again most likely because after the rsync the dovecot-uidvalidity changes. How do I prevent it from changing?

Thanks in advance!

darkimmortal

Why would you have POP3 enabled, let alone use it, in [insert current year]?

Vita

@darkimmortal said:
Why would you have POP3 enabled, let alone use it, in [insert current year]?

Well IMAP is enabled also, but POP3 a lot of people still use, so I can't just disable it. That is the thing.

Levi

Reindex or import those transfered emails. Use doveadm.

Vita

@LTniger said:
Reindex or import those transfered emails. Use doveadm.

The thing is that mails themselves are not moved at all, only control and index files to the new locations.
Similar to this:
https://www.tablix.org/~avian/blog/archives/2016/12/moving_dovecot_indexes_and_control_files/

For IMAP this is ok nothing is changed but for POP3 mails get new UIDL and get redownloaded.

Levi

If you move mail, and the new mail server generates UIDL lists differently then your mail program will think everything has to be downloaded again. To prevent this, always ensure the format of the UIDL is the same.

Few suggestions:

• You can switch all mailboxes to IMAP prior to transfer (POP3 is shit anyway)
• Prior to tranfser delete old mails (transfer will be faster)
• You may use tool like https://imapsync.lamiral.info/ if you transfer massive amounts of mailboxes

Vita

@LTniger The thing is that the migration is done on the same server (so same UIDL format), I just have to move control, index files to the new locations.

This is achieved by setting user query for dovecot in SQL so the mail column contains:

maildir:/nfs/users/userX
:INDEX=/nfs/dovecot/index/userX
:CONTROL=/nfs/dovecot/control/userX

The other thing is that I can't turn off POP3 ( I know it's shit, I would gladly switch, but it's not possible at the moment ). Since IMAP and POP3 are enabled mail is kept on the server and not removed, because some of the users use POP3 and IMAP (Webmail).

I don't actually need to transfer the emails themselves only the CONTROL, INDEX files and SIEVE files. Since those are from now on (for new users) created at those newly specified locations there is not a problem with new users, just the legacy ones that use POP3.

I can't seem to figure out why these files change after the moving of CONTROL path to the new one. If I migrate all other files and override the paths for INDEX and home everything is ok, but after I change it for CONTROL files it recreates the following files and increments the number:

-rw-------  1 virtual virtual    8 Feb  7 12:46 dovecot-uidvalidity
-r--r--r--  1 virtual virtual    0 Feb  7 11:36 dovecot-uidvalidity.5c5c0a2c

Zerpy

Maybe use symlinks and have legacy accounts have INDEX and CONTROL point to the old location, but in fact it's just symlinking.

Might not be super optimal, but at least it shouldn't redownload.

Vita

Sorry for bumping this thread, but I found a solution so maybe someone with similar problem would appreciate the answer. It was in front of me the whole time, but I just didn't notice it.

The issue with getting whole mailbox re-downloaded is because the control dovecot-uidlist file that was moved to separate location /nfs/dovecot/control/userX should have been moved to /nfs/dovecot/control/userX/.INBOX.

In the original structure .INBOX folder was not present because everything was in the same folder. If you override the CONTROL path you need to create .INBOX folder and put the control files in there in order for them not to get recreated.

Hope this helps someone, and this may be closed by a moderator, or maybe left open if someone has some questions so I can maybe share my experience.

Thanks to everyone who tried to help!