Running your own mail server sounds simple until you actually try doing it in production.

I recently set up a Mailcow instance on a small VPS and quickly discovered a bunch of real-world problems that most tutorials ignore:

  • SMTP ports blocked by VPS providers
  • Gmail rejecting mail due to missing PTR records
  • SSL certificate mismatches between host and containers
  • Dockerized mail services failing silently
  • High memory usage on smaller VPS instances
  • Reverse proxy confusion with external Nginx

After a lot of debugging, retries, and log reading, I ended up with a production-ready setup that works reliably.

In this guide, I’ll walk through exactly how I configured Mailcow using:

  • Dockerized Mailcow
  • External Nginx reverse proxy
  • LetsEncrypt SSL
  • Cloudflare DNS
  • SMTP on port 2525
  • Automatic certificate syncing
  • Low-memory VPS optimization

This is not a “copy random commands and pray” tutorial.

This is the setup I’d confidently deploy again.

Why I Chose Mailcow

There are several mail server solutions available, but I wanted something that:

  • Works in production
  • Has a decent UI
  • Supports DKIM, SPF, DMARC
  • Handles mailbox management cleanly
  • Runs inside Docker
  • Has active maintenance

That led me to:

Mailcow

Mailcow bundles together:

  • Postfix (SMTP)
  • Dovecot (IMAP/POP)
  • Rspamd (spam filtering)
  • DKIM support
  • SOGo webmail
  • Admin dashboard
  • Fail2Ban
  • Docker orchestration

Instead of manually configuring ten different services, Mailcow gives you a clean platform.

My Architecture

Before touching commands, let’s understand the architecture.

I intentionally avoided exposing Mailcow directly to the internet.

Instead, I placed Nginx in front.

Internet
   │
   ▼
Cloudflare DNS
   │
   ▼
Nginx (Host Machine)
   │ Reverse Proxy
   ▼
Mailcow Docker Stack
 ├── Postfix
 ├── Dovecot
 ├── Rspamd
 ├── SOGo
 └── Admin UI

Why?

Because I wanted:

  • Centralized SSL management
  • Certbot on host machine
  • Better flexibility
  • Easier debugging
  • Multiple apps on one server

This also prevents Mailcow from managing certificates itself.

Instead:

SKIP_LETS_ENCRYPT=y

Which means:

I become responsible for certificate syncing.

That part becomes important later.

VPS Requirements

You do not need a monster server.

My recommendation:

Resource Minimum
CPU 2 vCPU
RAM 2 GB
Storage 25+ GB SSD
Swap 2 GB

Mail servers spike memory usage occasionally.

If your VPS is small, swap becomes extremely important.

Step 1: Set the Server Hostname

The first thing I do on any mail server is set a proper hostname.

Bad hostname setup causes subtle email problems later.

Run:

sudo hostnamectl set-hostname mail.example.com

Verify:

hostnamectl status

Expected output:

Static hostname: mail.example.com

Your hostname should match your mail subdomain.

For example:

mail.example.com

Not:

ubuntu-server
localhost
vps01

This matters for mail reputation.

Step 2: Improve Shell Prompt Visibility

This is optional, but useful.

I prefer seeing the full hostname inside my shell.

Edit:

nano ~/.bashrc

Find:

PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '

Change:

\h

to:

\H

Example:

PS1='${debian_chroot:+($debian_chroot)}\u@\H:\w\$ '

Reload:

source ~/.bashrc

Now your shell becomes:

root@mail.example.com:~#

When managing infrastructure, tiny quality-of-life improvements matter.

Step 3: Add Swap (Highly Recommended)

If you’re deploying Mailcow on a small VPS, do not skip swap.

Mail services can briefly spike memory.

Without swap:

  • Docker containers crash
  • Databases restart
  • OOM killer terminates services

I create a 2 GB swap file.

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Persist after reboot:

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Verify:

free -h

Expected:

Swap: 2.0Gi

Step 4: Install Mailcow

I install Mailcow under /opt.

Move there:

cd /opt

Clone repository:

sudo git clone https://github.com/mailcow/mailcow-dockerized

Enter project:

cd mailcow-dockerized

Generate configuration:

sudo ./generate_config.sh

It asks:

Mail server hostname (FQDN):

Enter:

mail.example.com

This generates:

mailcow.conf

Step 5: Configure Mailcow for Reverse Proxy Mode

By default, Mailcow expects to handle web traffic directly.

I wanted Nginx in front.

So I changed Mailcow to internal-only ports.

Open:

nano /opt/mailcow-dockerized/mailcow.conf

Update:

HTTP_PORT=8080
HTTPS_PORT=8443

HTTP_BIND=127.0.0.1
HTTPS_BIND=127.0.0.1

SKIP_LETS_ENCRYPT=y
REDIRECT_HTTP_TO_HTTPS=n

SKIP_CLAMAV=y

Let’s break this down.

Internal Binding

HTTP_BIND=127.0.0.1
HTTPS_BIND=127.0.0.1

Mailcow is only accessible locally.

That means external users cannot bypass Nginx.

Good for security.

Disable Mailcow SSL

SKIP_LETS_ENCRYPT=y

I already manage SSL via Certbot.

Running two certificate systems becomes messy fast.

Disable ClamAV

SKIP_CLAMAV=y

On smaller VPS instances, ClamAV can consume noticeable memory.

For a low-memory deployment, this is a practical tradeoff.

Step 6: Why I Changed SMTP to Port 2525

This one took me hours to debug.

Many VPS providers block:

25
465
587

Why?

Spam prevention.

I was seeing mail failures because outbound SMTP traffic was silently blocked.

So instead of fighting provider restrictions, I exposed:

2525

Usually:

unblocked

This becomes useful for SMTP relay and testing.

Edit:

/opt/mailcow-dockerized/docker-compose.yml

Find:

postfix-mailcow:

Update ports:

ports:
  - "25:25"
  - "465:465"
  - "587:587"
  - "2525:2525"

Now Mailcow also listens on:

2525

This saved me a lot of pain.

Important: port 2525 is not a magical replacement for email delivery. Many providers still expect standard SMTP ports. I mainly use it when standard ports are blocked or for testing/debugging flows.

Step 7: Configure Nginx Reverse Proxy

Next, I configured Nginx on the host machine.

Create:

sudo nano /etc/nginx/sites-available/mail.example.com

Add:

server {
    listen 80;
    server_name mail.example.com;

    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name mail.example.com;

    ssl_certificate /etc/letsencrypt/live/mail.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mail.example.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:8080;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 600;
        proxy_read_timeout 600;
    }
}

Why proxy?

Because:

  • SSL stays centralized
  • Easier certificate renewal
  • Cleaner routing
  • Better observability

Enable site:

sudo ln -sf /etc/nginx/sites-available/mail.example.com /etc/nginx/sites-enabled/

Remove default:

sudo rm -f /etc/nginx/sites-enabled/default

Test:

sudo nginx -t

Reload:

sudo systemctl reload nginx

At this point:

https://mail.example.com

should route to Mailcow.

Step 8: Start Mailcow

Finally:

cd /opt/mailcow-dockerized
docker compose up -d

Check containers:

docker compose ps

You should see services like:

postfix-mailcow
dovecot-mailcow
nginx-mailcow
mysql-mailcow
rspamd-mailcow

If something fails:

docker compose logs -f

This command alone saved me hours of debugging.