Run Your Own Mail Server with Postfix & Dovecot

Self-hosting email is the most rewarding and most punishing self-hosted service you can run. Rewarding because you control your most important communication channel; punishing because the modern email ecosystem is hostile to small senders by default. Gmail, Outlook, and Yahoo all use reputation systems that treat any new sending IP as suspicious until proven otherwise — and proving otherwise takes weeks. This guide walks through a working Postfix + Dovecot setup with all the authentication records (SPF, DKIM, DMARC) the big providers require to even consider your mail. Plan an hour of focused work plus a week of reputation-building before relying on it.

Step 1: Prerequisites and DNS

You need a domain you control and a VPS with a clean IP address (not on any DNSBL — check at mxtoolbox.com). Set up the following DNS records before doing anything on the server:

; A record for the mail server itself
mail.example.com.    IN  A      203.0.113.10

; MX record pointing email to that server
example.com.         IN  MX  10 mail.example.com.

; PTR (reverse DNS) — set this through your VPS provider's panel,
; not your DNS host. Must resolve 203.0.113.10 -> mail.example.com

The PTR record is the one most setups skip. Without it, Gmail rejects your mail at the SMTP transaction with "Our system has detected an unusual rate of unsolicited mail." Most VPS panels have a "reverse DNS" or "PTR" field on the IP address page. Set it to your mail.example.com hostname, then verify with dig -x 203.0.113.10.

Step 2: Get a Let's Encrypt cert for the mail subdomain

You need TLS for both SMTP submission (587) and IMAP (993). Use certbot in standalone mode if you don't have a webserver running on this host:

sudo apt update
sudo apt install -y certbot
sudo certbot certonly --standalone -d mail.example.com

Certs land in /etc/letsencrypt/live/mail.example.com/. Set up a deploy hook so Postfix and Dovecot reload when certs renew:

sudo tee /etc/letsencrypt/renewal-hooks/deploy/reload-mail.sh > /dev/null <<'EOF'
#!/bin/bash
systemctl reload postfix dovecot
EOF
sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-mail.sh

Step 3: Install and configure Postfix

sudo apt install -y postfix postfix-pcre
# When the dialog appears, choose "Internet Site"
# and enter your domain (example.com) as the system mail name

Edit /etc/postfix/main.cf. Replace the relevant sections with these values (substituting your domain everywhere):

myhostname = mail.example.com
mydomain = example.com
myorigin = $mydomain
inet_interfaces = all
inet_protocols = ipv4
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain
home_mailbox = Maildir/

# TLS for incoming (port 25) — opportunistic
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.example.com/privkey.pem
smtpd_tls_security_level = may
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache

# TLS for outgoing
smtp_tls_security_level = may
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_tls_loglevel = 1

# Submission (port 587) — require auth + TLS
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
smtpd_sasl_local_domain = $myhostname

# Restrictions — block open relay, require auth for relay
smtpd_relay_restrictions =
  permit_mynetworks
  permit_sasl_authenticated
  reject_unauth_destination
smtpd_recipient_restrictions =
  permit_mynetworks
  permit_sasl_authenticated
  reject_unauth_destination
  reject_invalid_helo_hostname
  reject_non_fqdn_helo_hostname
  reject_unknown_recipient_domain

Then enable the submission service in /etc/postfix/master.cf — find the submission line and uncomment/edit it to look like this:

submission inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject

sudo systemctl restart postfix
sudo systemctl status postfix

Step 4: Install and configure Dovecot

Dovecot handles IMAP (so you can read mail from a client) and SASL auth (so Postfix can verify users when they submit mail).

sudo apt install -y dovecot-core dovecot-imapd

Edit /etc/dovecot/conf.d/10-mail.conf:

mail_location = maildir:~/Maildir

Edit /etc/dovecot/conf.d/10-auth.conf:

disable_plaintext_auth = yes
auth_mechanisms = plain login

Edit /etc/dovecot/conf.d/10-master.conf — find the service auth block and add the Postfix listener:

service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0660
    user = postfix
    group = postfix
  }
}

Edit /etc/dovecot/conf.d/10-ssl.conf:

ssl = required
ssl_cert = </etc/letsencrypt/live/mail.example.com/fullchain.pem
ssl_key  = </etc/letsencrypt/live/mail.example.com/privkey.pem
ssl_min_protocol = TLSv1.2

sudo systemctl restart dovecot

# Create a user account for testing
sudo adduser youruser
# Set a strong password — this is the SMTP/IMAP password

Step 5: Set up DKIM with OpenDKIM

DKIM cryptographically signs outgoing mail with a private key whose public counterpart is published in DNS. Without DKIM, Gmail and Outlook will treat your mail as suspicious or reject it outright.

sudo apt install -y opendkim opendkim-tools
sudo mkdir -p /etc/opendkim/keys/example.com
sudo cd /etc/opendkim/keys/example.com
sudo opendkim-genkey -s default -d example.com
sudo chown opendkim:opendkim default.private
sudo chmod 600 default.private
cat default.txt   # this is what you'll publish to DNS

Configure /etc/opendkim.conf:

Syslog                  yes
UMask                   002
Domain                  example.com
KeyFile                 /etc/opendkim/keys/example.com/default.private
Selector                default
Mode                    sv
Canonicalization        relaxed/relaxed
Socket                  inet:8891@localhost
PidFile                 /var/run/opendkim/opendkim.pid

Tell Postfix to use the milter — append to /etc/postfix/main.cf:

milter_protocol = 6
milter_default_action = accept
smtpd_milters = inet:localhost:8891
non_smtpd_milters = inet:localhost:8891

sudo systemctl restart opendkim postfix

Step 6: Add SPF and DMARC DNS records

These are TXT records in your DNS. Add them now — without SPF and DMARC, Gmail will be hostile to your mail no matter how perfect everything else is.

; SPF — authorize the mail server's IP to send for your domain
example.com.    IN  TXT  "v=spf1 mx -all"

; DKIM — paste the contents of default.txt from Step 5
default._domainkey.example.com.  IN  TXT  "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3..."

; DMARC — start in monitoring mode (p=none), tighten later
_dmarc.example.com.  IN  TXT  "v=DMARC1; p=none; rua=mailto:dmarc@example.com; ruf=mailto:dmarc@example.com; fo=1"

The p=none on DMARC means "tell receivers to report on what's failing, but don't reject anything yet." Run in monitoring mode for at least two weeks while you watch the reports, then tighten to p=quarantine and eventually p=reject.

Step 7: Test deliverability

Three tests, in order:

1. Send to mail-tester.com. Send a message from your new account to the address it gives you, then check the score. Anything below 9/10 has a fixable issue — the report tells you exactly what.

2. Send to a Gmail account you own. If it lands in Gmail, open the message, click the three-dot menu, "Show original." You should see SPF: PASS, DKIM: PASS, DMARC: PASS. Any FAIL is a real problem.

3. Check blacklists. Go to mxtoolbox.com/blacklists.aspx and check your IP. If you're listed, request delisting from each blacklist (most have a self-service form).

Step 8: Spam filtering with rspamd

rspamd is the modern Postfix-friendly spam filter (replacing the older SpamAssassin pattern for new installs). It also handles DKIM signing and verification, replacing OpenDKIM if you'd rather have one less daemon. For an existing OpenDKIM setup, run rspamd alongside in scan-only mode.

sudo apt install -y rspamd redis-server
sudo systemctl enable --now rspamd redis-server

# Set the controller password
sudo rspamadm pw
# Paste output into /etc/rspamd/local.d/worker-controller.inc:
#   password = "$2$..."

# Enable in Postfix — append to /etc/postfix/main.cf:
sudo tee -a /etc/postfix/main.cf <<'EOF'
smtpd_milters = inet:localhost:8891 inet:localhost:11332
EOF

sudo systemctl restart postfix rspamd

The rspamd web UI lives at http://localhost:11334 (tunnel through SSH to view; never expose it publicly). It shows score histograms, spam learning stats, and lets you manually train ham/spam.

Common issues and fixes

Port 25 shows closed from outside — your provider is blocking it. Open a ticket asking them to unblock outbound 25; most will after a quick check. If they refuse, you'll need to use a smarthost like Mailgun or AWS SES (which defeats most of the point of self-hosting).

Gmail still sends to spam after everything passes — IP reputation. Brand new IPs sit in a probation period. Send small amounts of legitimate mail (5–10 messages a day for a week), have recipients mark them "Not Spam" and add you to contacts. Don't try to "warm" the IP with throwaway sends — that backfires.

relay access denied from your own client — your mail client isn't authenticating. Check that you're using port 587 (not 25) and the encryption type is "STARTTLS" (not "SSL/TLS" — that's for port 465 if you've enabled smtps).

DKIM signature missing on outgoing mail — check sudo journalctl -u opendkim for permission errors on the key file. The key must be readable by opendkim. Also confirm the milter is listed in postfix/main.cf and Postfix has been restarted since the change.

Mail goes out fine but never arrives — check /var/log/mail.log for the queue ID, then postqueue -p for the deferred queue. If it says "deferred (host x.x.x.x said: 550 5.7.1 ...)", the receiving server is rejecting your mail, and the response usually says why. If it says "Connection timed out", port 25 outbound is blocked.

FAQ