Host a Personal OpenBSD Mail Server

Setting up email is more rewarding, though a little more effort, than hosting your own website. While a web server for the real world only requires launching two programs (the web server itself, and the ACME client to maintain certificates for HTTPS), a mail server ready to exchange mail with tech giants requires wiring up four tools together:

• OpenSMTPD: to receive mail from your local mail program (like mutt or Thunderbird), and to exchange mail between other mail servers.
• ACME client: to manage the cert that certifies to other mail servers that you’re legitimate. This is the same process used to enable HTTPS.
• Dovecot: to deliver mail to your local mail program for you to read.
• Rspamd: to filter spam. The program also signs all outgoing mail with a DKIM signature, which verifies that your server is the one that created it.

A mail server will also require a few DNS records to tell the world that your server and the mail it sends is legitimate.

Preparation

OpenBSD provides OpenSMTPD and acme-client as part of its base system, but everything else needs to be installed. Connect to your server and install all required software:

$ doas pkg_add rspamd opensmtpd-filter-rspamd dovecot

Setup DNS and rDNS records

Foreign mail servers need a few bits of information from DNS:

• What hosts are allowed to create your mail (SPF)
• A public key that verifies your mail came from you (DKIM)
• What to do with mail that fails these security checks (DMARC)
• Which hosts to communicate with to receive your mail (MX)

Additionally, you need to configure your VPS to set an rDNS record to point your IP address back to your hostname.

It could take form a few hours up to a day for DNS records to make it way through the Internet’s DNS servers, so we’ll add all the records today, and set up all the software afterwards.

Create the DKIM Key

Installing Rspamd will create the system user _rspamd, which we’ll set as the owner of the private DKIM key we’ll create.

Create the private key. I’ll be naming the key after the domain it’ll be attached to and the date the key would be published (15th January 2022, so example.org.20220118.key):

$ cd /etc/mail
$ doas mkdir -m 700 dkim
$ doas openssl genrsa -out dkim/example.org.20220118.key 2048
$ doas chmod 400 dkim/example.org.20220118.key
$ doas chown -R _rspamd:_rspamd dkim

• mkdir creates the directory /etc/mail/dkim, and sets the mode so only its owner can access it.
• openssl genrsa creates the private key.
• chmod 400 limits access to the private key to the owner, who is only able to read it.
• chown -R _rspamd:_rspamd sets the owner of /etc/mail/dkim and all its contents (only the private key in this case) to the system account and group _dkim.

Create the public key, and keep its contents to the side. It will be the p field of your DKIM DNS entry:

$ doas openssl rsa -in /etc/mail/dkim/example.org.20220118.key -pubout | grep -v ^--- | tr -d '\n'
writing RSA key
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0wwm......TNOPwG1//++gfl/hwIDAQAB

openssl rsa ... -pubout generates the public key, and feeds it to grep and tr, which formats the key into a single-line string of text.

Setup DNS

Add these 4 DNS records:

example.org.                    MX      "mail.example.org"    10
20220118._domainkey.example.org TXT     "v=DKIM1; k=rsa; p=MIIBIj......hwIDAQAB"
example.org.                    TXT     "v=spf1 mx -all"
_dmarc.example.org.             TXT     "v=DMARC1; p=reject; rua=mailto:reports@example.org; fo=1"

• The DKIM record provides the world the public key needed to verify your mail’s DKIM signatures.
• The SPF policy mx -all tells the world to trust all mail claiming to come from your domain only if the source IP address matches the hostname of your MX record. All other IP addresses are blocked.
• The DMARC policy p=reject; rua=mailto:reports@example.org; fo=1 tells the world to reject all mail that doesn’t pass the DKIM and SPF tests, and to send reports about received mail to reports@example.org.

Setup rDNS

Additionally, add a reverse DNS record to your server that matches your domains’ MX records. How to do this depends entirely on your ISP (or VPS provider), but for the Vultr server I’m using, on the Products dashboard:

1. Click on your VPS instance
2. Visit the Settings menu
3. Visit the IPv4 tab
4. On the “Reverse DNS” column of the table, replace the given domain with mail.example.org.
5. Visit the IPv6 tab, and in the Reverse DNS section, add a record with your server’s IPv6 address and the same domain as the last tab.

Finally, after both DNS and rDNS records are in place, take a small break. Every few hours, you can check to see if your records have propagated through with DNS lookup.

Open mail-related ports on the firewall

If you have Packet Filter set up, add this line to /etc/pf.conf:

pass in on egress proto tcp to port {smtp imaps pop3s smtps msa}

• Smtpd listens on port smtp to accept mail from the world, and smtps to accept mail from its users.
• Dovecot listens on ports imaps and pop3s to deliver mail to its users.

Check that pf.conf is configured properly and load it:

$ doas pfctl -nf /etc/pf.conf
$ doas pfctl -f /etc/pf.conf

Configure ACME client

Just like with a web server, mail servers encrypt their communication with SSL/TLS. They need a cert, so we can set up OpenBSD’s ACME client to do this for us.

Add your domain to /etc/acme-client.conf, and add Let’s Encrypt as an authority:

authority letsencrypt {
        api url "https://acme-v02.api.letsencrypt.org/directory"
        account key "/etc/acme/letsencrypt-privkey.pem"
}

domain mail.example.org {
        domain key "/etc/ssl/private/mail.example.org.key"
        domain certificate "/etc/ssl/mail.example.org.cert"
        domain full chain certificate "/etc/ssl/mail.example.org.fullchain.pem"
        sign with letsencrypt
}

Run ACME client to generate the certificate:

$ doas acme-client -v mail.example.org

Finally, add the command to /etc/weekly.local to renew the certificate automatically over time:

acme-client -v mail.example.org

Setup Rspamd

Rspamd’s primary goal is to quickly filter out spam coming from the outside world. It can also be configured to sign outgoing messages with a DKIM signature, so we’ll use this program to kill two birds with one stone. Internet mail servers can check this signature with the public key provided in our DKIM DNS record to verify that our messages were not forged by an impersonator.

Fix /etc/resolv.conf if needed

Before setting up Rspamd, I found I needed to do this nasty hack to Vultr’s default installation of OpenBSD. Check to see if the line lookup file bind is found in /etc/resolv.conf.

$ cat /etc/resolv.conf

If the line does not exist, and you see lines that end with # resolvd: vio0 (or the name of some other network interface), then add the line to the end of both /etc/resolv.conf.tail and resolv.conf:

lookup file bind

Rspamd is configured by default to bind to localhost and listen on a number of ports. The file /etc/hosts point localhost to the IP address 127.0.0.1 but if no lines that start with lookup are in resolv.conf, it does a DNS lookup query first. I’ve found anecdotally that their DNS servers interpret localhost to localhost.net (???), and return two of Cloudflare’s IP addresses. Since Rspamd can’t listen “from” an outside IP address, it crashes on startup.

Configure Rspamd to enable DKIM signing

With that out of the way, create the file /etc/rspamd/local.d/dkim_signing.conf and add these settings:

allow_username_mismatch = true;

domain {
        example.org {
                selector = "20220118";
                path = "/etc/mail/dkim/example.org.20220118.key";
        }
}

Start Rspamd

Enable and start Rspamd:

$ doas rcctl enable redis rspamd
$ doas rcctl start redis rspamd
redis(ok)
rspamd(ok)
$ doas rcctl check rspamd
rspamd(ok)

Redis is a data store used by Rspamd and installed as a dependency. We didn’t need to configure it; Rspamd knows how to use it as-is.

rcctl check rspamd reports whether Rspamd is still running after it was moved to the background. I’ve had the program fail shortly after starting up if I haven’t configured resolv.conf, so I’ve made it a habit of double-checking whenever I restart it.

Setup Dovecot

Create a Diffie-Hellman key used by Dovecot to negotiate a shared key used to open encrypted communications with its users. This takes a long time, so I recommend opening a fresh remote terminal, so that it can run on the side while setting up Dovecot:

$ cd /etc/mail
$ doas openssl dhparam -out dh.pem 4096
$ doas chown _dovecot:_dovecot dh.pem
$ doas chmod 400 dh.pem

Dovecot may open many more files at the same time than OpenBSD’s conservative restrictions allow by default. Extend Dovecot’s open-file privileges by adding this paragraph to /etc/login.conf:

dovecot:\
      :openfiles-cur=1024:\
      :openfiles-max=2048:\
      :tc=daemon:

By default, Dovecot has a “main” config file in /etc/dovecot/dovecot.conf, which loads in both /etc/dovecot/local.conf and all conf files in /etc/dovecot/local.d. It’s easier to overwrite the main file though, so move it somewhere safe:

$ cd /etc/dovecot
$ doas mv dovecot.conf dovecot.conf.bkup

Create the file /etc/dovecot/dovecot.conf:

# ---SSL---
# Require TLSv1.2+ all the time; prefer the server's ciphers and provide key, cert, and dh params
ssl = required
ssl_min_protocol = TLSv1.2
ssl_prefer_server_ciphers = yes

ssl_cert = </etc/ssl/mail.example.org.fullchain.pem
ssl_key = </etc/ssl/private/mail.example.org.key
ssl_dh = </etc/mail/dh.pem


# ---Auth---
# Accept plaintext passwords to authenticate against real BSD accounts
auth_mechanisms = plain
auth_username_format = %Ln
userdb {
        driver = passwd
}

passdb {
        driver = bsdauth
}

# The "login" mechanism is outdated, but still used by old versions of Outlook
# and some Microsoft phones. Enable it here if you plan to support these.
#auth_mechanisms = $auth_mechanisms login


# ---Mailbox structure---
# Outside the inbox, also provide Archive, Drafts, Junk,
# Sent, and Trash boxes. Mark them all as its special behavior as defined in
# RFC6154.2 . Set all
# boxes to create + subscribe to automatically when a mail client connects to us.
mail_location = maildir:~/Mail:INBOX=~/Mail/Inbox:LAYOUT=fs
namespace inbox {
        inbox = yes
        mailbox Archive {
                auto = subscribe
                special_use = \Archive
        }
        mailbox Drafts {
                auto = subscribe
                special_use = \Drafts
        }
        mailbox Junk {
                auto = subscribe
                autoexpunge = 30d
                special_use = \Junk
        }
        mailbox Sent {
                auto = subscribe
                special_use = \Sent
        }
        mailbox Trash {
                auto = subscribe
                special_use = \Trash
        }
}

# Protocols
protocols = imap pop3

Start Dovecot

Check, start, and enable Dovecot:

$ doas doveconf -n
$ doas rcctl enable dovecot
$ doas rcctl start dovecot
dovecot(OK)

Setup OpenSMTPD

OpenSMTPD handles the main job of exchanging messages between its users and foreign mail servers. OpenBSD provides it as a first-party program, and is enabled by default to transfer messages locally between system users. We want to open this mail server to accept envelopes from the Internet.

Configure OpenSMTPD

Replace /etc/mail/smtpd.conf with:

# Smtpd uses TLS, so point it to the cert and key we made with the ACME client.
pki mail.example.org cert "/etc/ssl/mail.example.org.fullchain.pem"
pki mail.example.org key "/etc/ssl/private/mail.example.org.key"

table aliases file:/etc/mail/aliases

# A good number of outgoing spam hosts come from compromised residential machines.
# Most of these machines live behind a dynamic IP address. This filter rejects all
# mail whose reverse DNS record looks like it came from a dynamic IP address.
filter "no_dyndns" phase connect match rdns regex { '.*\.dyn\..*', '.*\.dsl\..*' } \
	disconnect "550 no residential connections"

# Reject all mail whose reverse DNS record doesn't have a valid reverse DNS
# record, or whose records doesn't loop back to itself.
filter "no_rdns" phase connect match !rdns \
	disconnect "550 mailserver failed rDNS check"
filter "no_fcrdns" phase connect match !fcrdns \
	disconnect "550 mailserver failed FCrDNS check"

# Rspamd detects and marks spam for incoming mail, and
# signs DKIM signatures for outgoing mail.
filter "rspamd" proc-exec "filter-rspamd"

# ---Inbound Mail---

# Listen on the SMTP port. Ideally this would always be encrypted, but not all servers
# support that. Frustratingly, I've had services send mail unencrypted with my
# account name and password in plaintext...
listen on egress tls pki mail.example.org auth-optional \
        filter { "no_dyndns", "no_rdns", "no_fcrdns", "rspamd" }

action "inbound" maildir "~/Mail/Inbox" alias <aliases>

match from any for local action "inbound"

# ---Outbound Mail---

# Smtpd already listens on a unix socket by itself -- explicitly listing it here
# lets us add our filters for outgoing mail.
listen on socket	filter "rspamd"

# Listen on the MSA and SMTPS ports, two avenues for users to submit their
# mail. MSA starts unencrypted, but we force it to upgrade to TLS. SMTPS begins
# TLS-encrypted from the start.
listen on egress	port msa tls-require pki mail.example.org auth filter "rspamd"
listen on egress	smtps pki mail.example.org auth filter "rspamd"

action "outbound" relay

match from local for any action "outbound"
match from auth for any action "outbound"

Start Smtpd

Check the configuration is OK, and restart OpenSMTPD:

$ doas smtpd -n
$ doas rcctl restart smtpd
smtpd(ok)
smtpd(ok)

Test OpenSMTPD is working

AdminSystem Software Ltd. offers a free DKIM validator you can use to verify you can send email with valid DKIM, SPF, DMARC, and PTR (reverse DNS) settings.

Visit the page and start the test. It should give you an email to send to, such as test-abcd1234@appmaildev.com. Use sendmail to send a message to the service:

$ sendmail test-abcd1234@appmaildev.com << EOF
> Subject: DKIM
>
> DKIM
> EOF

>> EOF is a feature of OpenBSD’s shell, which feeds in all input given to the command, until it reaches a single “EOF”.

If Smtpd and your domain’s DNS records are properly set up, the DKIM validator should give these results:

• SPF: Pass
• DKIM: Pass
• DMARC: Pass
• DomainKey: None (this is an older standard that’s replaced with DKIM)
• PTR: ExistsRecord

Test Rspamd is filtering spam messages with GTUBE

The GTUBE is a string of text that tests anti-spam systems, like Rspam or SpamAssassin. Apache hosts an example message that holds the GTUBE string.

To test Rspamd, download the GTUBE on the server and pass it to sendmail:

$ wget https://spamassassin.apache.org/gtube/gtube.txt
$ sendmail root@example.org < gtube.txt
sendmail: command failed: 550 Gtube pattern

If wget doesn’t exist, install it via pkg_add wget. The failure message 550 Gtube pattern signals that Smtpd rejected the mail.

Create an email account

Since we haven’t configured virtual users, Smtpd tightly couples system accounts with email accounts by default. Create a new system account:

$ doas adduser johndoe

Smtpd will transfer all messages marked for johndoe@example.org to this new system account.

Log on with Thunderbird

Any mail user agent can do, but Thunderbird is free, open-source, mostly intuitive, and available on Windows, Mac, Linux, and various BSDs. You should be able to follow the main concepts with any similar program.

1. On the Thunderbird dashboard, under “Set Up Another Account”, click on Email.
2. Provide your full name, your email address (johndoe@example.org), and password.
3. Thunderbird will likely fail to autoconfigure your account, since they don’t recognize it. Manually fill in these settings:
   • Incoming Server
     • Protocol: IMAP
     • Hostname: mail.example.org
     • Port: 993
     • Connection security: SSL/TLS
     • Authentication method: Norml password (it’s encrypted already through SSL)
     • Username: johndoe (drop the @example.org)
   • Outgoing Server
     • Hostname: mail.example.org
     • Port: 465
     • Connection security: SSL/TLS
     • Authentication method: Normal password
     • Username: johndoe
4. Click “Done”.

Thunderbird should successfully log in. Wait a few seconds for your folders to populate to signify that incoming mail (IMAPS through Dovecot) is working. You can test if outgoing mail (SMTPS through Smtpd) is working by visting the DKIM test service like before.

Summary

You’re now hosting a mail server facing the outside world, set up to protect against spam, spoofing attacks, and eavesdroppers. Here’s a technical summary all components involved:

• MX records tell foreign mail servers that email intended for @example.org should be received by mail.example.org.
• Smtpd listens for connections through SMTP to receive mail from the world and deliver them to your account.
• Dovecot listens for connections through IMAP and POP3 to deliver mail from the mail server to your device.
• SSL certificates overall certify to foreign mail servers that they are opening encrypted communications to the correct host.
• SPF tells the world that only mail.example.org is allowed to create mail marked from @example.org.
• DKIM protects the world from mail pretending to be created by your computer.
• DMARC tells the world to reject mail that they fail to verify as yours through DKIM and SPF, and to report them to reports@example.org.

Appendix: Configuration notes

I used Rspamd to sign DKIM messages. Here are some of the alternatives I considered:

• dkimproxy’s latest version (1.4.1) was released in 2011 — over a decade ago. After a cursory look, I saw active issues on the Sourceforge forum, and no pages in the changelog or elsewhere that implied it was given a proper development freeze, or moved on to another project. filter-dkimsign is given as an example in smtpd.conf(5).
• It is also newer: it’s LICENSE for the latest version (0.5) is in 2019. However, I found personally after trying it out that it gives a bad DKIM signature as reported by the DKIM test tool. Also, another user named guidocella reports that they have had issues with the program crashing on special edge cases.
• Rspamd includes DKIM signing, spam filtering, and from what I can tell, a heck of a lot more. Since I want to filter my spam anyways, it makes sense to use it to handle DKIM signing as well. smtpd.conf(5) also lists filter-rspamd as an example.

Appendix: Troubleshooting Rspamd

If rspamd fails to start, you can usually check its logs at /var/log/rspamd/rspamd.log to find the reason why it crashed.

lookup file bind

Going Further

• Host a Backup Relay Mail Server on OpenBSD
• Host an OpenBSD Mail Server with Virtual Email Accounts

External Links

• Setting up an email server in 2020 with OpenSMTPD and Dovecot - Marcus Newman
• Setting up a mail server with OpenSMTPD, Dovecot and Rspamd - Gilles Chehade, maintainer and main developer of OpenSMTPD
• Setting Up Your Own Email Server With OpenBSD - Alex Useche