Running your own mail server sounds simple until you actually try doing it in production.
I recently set up a Mailcow instance on a small VPS and quickly discovered a bunch of real-world problems that most tutorials ignore:
- SMTP ports blocked by VPS providers
- Gmail rejecting mail due to missing PTR records
- SSL certificate mismatches between host and containers
- Dockerized mail services failing silently
- High memory usage on smaller VPS instances
- Reverse proxy confusion with external Nginx
After a lot of debugging, retries, and log reading, I ended up with a production-ready setup that works reliably.
In this guide, I’ll walk through exactly how I configured Mailcow using:
- Dockerized Mailcow
- External Nginx reverse proxy
- LetsEncrypt SSL
- Cloudflare DNS
- SMTP on port
2525 - Automatic certificate syncing
- Low-memory VPS optimization
This is not a “copy random commands and pray” tutorial.
This is the setup I’d confidently deploy again.
Why I Chose Mailcow
There are several mail server solutions available, but I wanted something that:
- Works in production
- Has a decent UI
- Supports DKIM, SPF, DMARC
- Handles mailbox management cleanly
- Runs inside Docker
- Has active maintenance
That led me to:
Mailcow bundles together:
- Postfix (SMTP)
- Dovecot (IMAP/POP)
- Rspamd (spam filtering)
- DKIM support
- SOGo webmail
- Admin dashboard
- Fail2Ban
- Docker orchestration
Instead of manually configuring ten different services, Mailcow gives you a clean platform.
My Architecture
Before touching commands, let’s understand the architecture.
I intentionally avoided exposing Mailcow directly to the internet.
Instead, I placed Nginx in front.
Internet
│
▼
Cloudflare DNS
│
▼
Nginx (Host Machine)
│ Reverse Proxy
▼
Mailcow Docker Stack
├── Postfix
├── Dovecot
├── Rspamd
├── SOGo
└── Admin UI
Why?
Because I wanted:
- Centralized SSL management
- Certbot on host machine
- Better flexibility
- Easier debugging
- Multiple apps on one server
This also prevents Mailcow from managing certificates itself.
Instead:
SKIP_LETS_ENCRYPT=y
Which means:
I become responsible for certificate syncing.
That part becomes important later.
VPS Requirements
You do not need a monster server.
My recommendation:
| Resource | Minimum |
|---|---|
| CPU | 2 vCPU |
| RAM | 2 GB |
| Storage | 25+ GB SSD |
| Swap | 2 GB |
Mail servers spike memory usage occasionally.
If your VPS is small, swap becomes extremely important.
Step 1: Set the Server Hostname
The first thing I do on any mail server is set a proper hostname.
Bad hostname setup causes subtle email problems later.
Run:
sudo hostnamectl set-hostname mail.example.com
Verify:
hostnamectl status
Expected output:
Static hostname: mail.example.com
Your hostname should match your mail subdomain.
For example:
mail.example.com
Not:
ubuntu-server
localhost
vps01
This matters for mail reputation.
Step 2: Improve Shell Prompt Visibility
This is optional, but useful.
I prefer seeing the full hostname inside my shell.
Edit:
nano ~/.bashrc
Find:
PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
Change:
\h
to:
\H
Example:
PS1='${debian_chroot:+($debian_chroot)}\u@\H:\w\$ '
Reload:
source ~/.bashrc
Now your shell becomes:
root@mail.example.com:~#
When managing infrastructure, tiny quality-of-life improvements matter.
Step 3: Add Swap (Highly Recommended)
If you’re deploying Mailcow on a small VPS, do not skip swap.
Mail services can briefly spike memory.
Without swap:
- Docker containers crash
- Databases restart
- OOM killer terminates services
I create a 2 GB swap file.
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
Persist after reboot:
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
Verify:
free -h
Expected:
Swap: 2.0Gi
Step 4: Install Mailcow
I install Mailcow under /opt.
Move there:
cd /opt
Clone repository:
sudo git clone https://github.com/mailcow/mailcow-dockerized
Enter project:
cd mailcow-dockerized
Generate configuration:
sudo ./generate_config.sh
It asks:
Mail server hostname (FQDN):
Enter:
mail.example.com
This generates:
mailcow.conf
Step 5: Configure Mailcow for Reverse Proxy Mode
By default, Mailcow expects to handle web traffic directly.
I wanted Nginx in front.
So I changed Mailcow to internal-only ports.
Open:
nano /opt/mailcow-dockerized/mailcow.conf
Update:
HTTP_PORT=8080
HTTPS_PORT=8443
HTTP_BIND=127.0.0.1
HTTPS_BIND=127.0.0.1
SKIP_LETS_ENCRYPT=y
REDIRECT_HTTP_TO_HTTPS=n
SKIP_CLAMAV=y
Let’s break this down.
Internal Binding
HTTP_BIND=127.0.0.1
HTTPS_BIND=127.0.0.1
Mailcow is only accessible locally.
That means external users cannot bypass Nginx.
Good for security.
Disable Mailcow SSL
SKIP_LETS_ENCRYPT=y
I already manage SSL via Certbot.
Running two certificate systems becomes messy fast.
Disable ClamAV
SKIP_CLAMAV=y
On smaller VPS instances, ClamAV can consume noticeable memory.
For a low-memory deployment, this is a practical tradeoff.
Step 6: Why I Changed SMTP to Port 2525
This one took me hours to debug.
Many VPS providers block:
25
465
587
Why?
Spam prevention.
I was seeing mail failures because outbound SMTP traffic was silently blocked.
So instead of fighting provider restrictions, I exposed:
2525
Usually:
unblocked
This becomes useful for SMTP relay and testing.
Edit:
/opt/mailcow-dockerized/docker-compose.yml
Find:
postfix-mailcow:
Update ports:
ports:
- "25:25"
- "465:465"
- "587:587"
- "2525:2525"
Now Mailcow also listens on:
2525
This saved me a lot of pain.
Important: port 2525 is not a magical replacement for email delivery. Many providers still expect standard SMTP ports. I mainly use it when standard ports are blocked or for testing/debugging flows.
Step 7: Configure Nginx Reverse Proxy
Next, I configured Nginx on the host machine.
Create:
sudo nano /etc/nginx/sites-available/mail.example.com
Add:
server {
listen 80;
server_name mail.example.com;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl;
server_name mail.example.com;
ssl_certificate /etc/letsencrypt/live/mail.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mail.example.com/privkey.pem;
location / {
proxy_pass http://127.0.0.1:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 600;
proxy_read_timeout 600;
}
}
Why proxy?
Because:
- SSL stays centralized
- Easier certificate renewal
- Cleaner routing
- Better observability
Enable site:
sudo ln -sf /etc/nginx/sites-available/mail.example.com /etc/nginx/sites-enabled/
Remove default:
sudo rm -f /etc/nginx/sites-enabled/default
Test:
sudo nginx -t
Reload:
sudo systemctl reload nginx
At this point:
https://mail.example.com
should route to Mailcow.
Step 8: Start Mailcow
Finally:
cd /opt/mailcow-dockerized
docker compose up -d
Check containers:
docker compose ps
You should see services like:
postfix-mailcow
dovecot-mailcow
nginx-mailcow
mysql-mailcow
rspamd-mailcow
If something fails:
docker compose logs -f
This command alone saved me hours of debugging.