Cheap VPS & Xen Server


Residential Proxy Network - Hourly & Monthly Packages

Full Mail Server Solution w/ Virtual Domains & Users (Debian Etch, Postfix, Mysql, Dovecot, DSpam, ClamAV, Postgrey, RBL)


This tutorial is Copyright (c) 2007 by Justin Refice. It is derived from various guides and original material, listed at the end of the document. You are free to use this tutorial under the Creative Commons license 2.5 or any later version.

I. Introduction

This guide describes how to set up a full email solution in Debian Linux (all code is from Debian Etch).  I was asked to design a secure, scalable, portable solution for a small company.  While the guide references many ‘servers’, the company only had 4 physical machines, Xen was used to virtualize the entire solution.  That particular aspect of the system is not discussed in this guide, although I will try to get it into the next revision.

Just a note on the server names used below: If it doesn’t need to be accessed by the internet, don’t let it be. Domain names ending in internal.example.com are internal NIC/IP Addresses… there is no way to access them directly from the internet, nor should there be.  Any server that ONLY has an internal.example.com domain name is a pure-internal server, and can’t be accessed directly from the internet.  All non-internal servers have two NICS (These can be two real NICs, or virtual).  The first NIC has access to the internet, and is strictly firewalled.  The second NIC has access to the internal network, and has a little less security as a result.  The details of how to setup these NICs are outside the scope of this document, but I may update it to include them in the future.

The general layout of the servers is:

Primary MX:
NIC1 = Insecure/Internet = mx-1.example.com
NIC2 = Secure/Intranet = mx-1.internal.example.com
MTA: Postfix
Greylist Filter: Postgrey

Secondary MX:
NIC1 = Insecure/Internet = mx-2.example.com
NIC2 = Secure/Intranet = mx-2.internal.example.com
MTA: Postfix
Greylist Filter: Postgrey

SMTP+TLS & IMAPS:
NIC1 = Insecure/Internet = secure-mail.example.com
NIC2 = Secure/Intranet = secure-mail.internal.example.com
MTA: Postfix (+TLS/SSL)
IMAP: Dovecot (IMAPS)

Mail Delivery Server: postman.internal.example.com
MTA (lmtp): DSPAM
Antivirus: ClamAV
IMAP: Dovecot

Database Server: sql-1.internal.example.com
MySQL

File Server: files-1.internal.example.com
NFS

Temporary Build Server: build.internal.example.com
<Various Tools>

Mail works in the following way:

Internet mail to your domains:

1. Mail comes in to Primary or Secondary MX on port 25
2. MX queries MySQL server to see if mail recipient & destination are valid:
a. Recipient is unauthorized – Mail is rejected (550 Error)
b. Recipient is authorized – Mail is is allowed to continue
3. MX checks greylist policy:
a. This is the first time email is tried – Mail is rejected (Retry)
b. This is not the first time email is tried – Mail is allowed to continue
4. MX checks for quota violations
a. The user’s quota is full – Mail is bounced
b. The user has room – Mail is delivered
5. MX Sends mail to Internal Delivery Server (via LMTP)
6. Internal Delivery Server checks for Virus/SPAM
a. This is SPAM – SPAM is marked, and given to LDA for delivery.
b. This is a virus – Mail is rejected
c. This is NOT SPAM and NOT VIRUS – Mail is given to LDA
7. LDA Delivers mail
a. The mail is marked as SPAM – Delivered to “SPAM” directory in Maildir
b. The mail is NOT marked as SPAM – Delivered to inbox.

Internet mail from your domains:

1. User initiates connection to SMTP Relay on port 25
2. SMTP Relay offers TLS:
a. User does not use TLS – Mail is rejected
b. User does use TLS – Session is is allowed to continue
3. SMTP Relay offers AUTH (PLAIN):
a. User does not authenticate/Fails Authentication – Mail is rejected
b. User does authenticate – Session completes as usual

Remote users access mail via IMAPS (Secure IMAP)

Local users access mail via IMAP

If the user detects a false positive SPAM detection, they forward the email to “ham-<username>@<domain>.<tld>”
If the user detects a false negative SPAM detection, they forward the email to “spam-<username>@<domain>.<tld>”

II. Important Notes

All this may be installed in either Debian 4.0 Etch or Ubuntu Feisty Fawn, since both systems are quite similar. Note however that there may be some minor issues if you use the default version of Dovecot and Postfix, but I will try to note them down for you when they arise.

If you are a Ubuntu user, note that I will not use “sudo” in front of every command. Instead, I will launch a root shell using the command “sudo -s”.

Installing software in Ubuntu & Debian is very easy, so whenever possible we’ll be using the build in apt-get utility. The less we have to build ourselves, the easier it is to maintain later.

So, let’s get started!

III. Installing and configuring MySQL Server (+PostFixAdmin Schema)

First off, we need to install the MySQL server on sql-1.internal.example.com. To make life easy, we’re also going to install the MySQL command line client. This can be achieved by typing the following at the prompt:

# apt-get install mysql-server mysql-client

After apt-get has done its installation mojo, you’re going to want to lock the root access to the mysql database. This is done using the mysqladmin tool.

# mysqladmin -u root password “mypassword”

NOTE: Change mypassword to your own secure password!!!

Now we’re going to create the database and users for Virtual Users & Domains. It’s important to note that Postfix (And Dovecot) only require SELECT access to this database. PostfixAdmin will require SELECT, UPDATE, and INSERT.

# mysql -uroot -p
Enter password:
mysql> CREATE DATABASE virtual_mail;
mysql> GRANT SELECT ON virtual_mail.* TO ‘vmail_user’@’mx-1.example.com’ IDENTIFIED BY ‘vmail_user_password’;
mysql> GRANT SELECT,UPDATE,INSERT ON virtual_mail.* TO ‘vmail_admin’@’mx-1.example.com’ IDENTIFIED BY ‘vmail_admin_password’;
mysql> GRANT SELECT ON virtual_mail.* TO ‘vmail_user’@’mx-2.example.com’ IDENTIFIED BY ‘vmail_user_password’;
mysql> GRANT SELECT,UPDATE,INSERT ON virtual_mail.* TO ‘vmail_admin’@’mx-2.example.com’ IDENTIFIED BY ‘vmail_admin_password’;
mysql> quit

NOTE: Change vmail_user_password and vmail_admin_password to your own secure passwords!!!

Now we need to download Postfix Admin. While the tool itself is not manditory for our configuration, it is very useful and the schema it uses are well thought out. Once you have downloaded the Postfix Admin distribution tarball, unpack it like this:

# tar xfvz postfixadmin-2.1.0.tgz

Inside the resulting directory, open postfixadmin-2.1.0/DATABASE_MYSQL.TXT with your favorite text editor. Remove all the lines under the “Postfix / MySQL” section. These lines create the initial database, which we’ve already done. For Postfix Admin 2.1.0 these are lines 26-39.

When complete, load the file into your existing MySQL table with the following command:

# mysql -uroot -p virtual_mail < postfixadmin-2.1.0/DATABASE_MYSQL.TXT

NOTE: If you get an error saying “Access denied for user ‘mail’@’localhost’ to database ‘mysql'”, then you didn’t comment out the lines properly. If you get no message after entering your password, then everything went fine.

IV. NFS File Share Server

A lot of servers are going to require access to our users Maildir folders, so in order to make life simple(r), we’re going to install them on an NFS mount. One of the benefits of Maildir is that it is compatible with NFS. Mailbox format on the other hand, would not be pretty. Luckily, setting up NFS in Debian is just as simple as setting up any other service.

Let’s start by installing the service on files-1.internal.example.com:

# apt-get install nfs-kernel-server nfs-common portmap

The NFS Exports (Shares) are controlled by the file /etc/exports. Each line begins with the absolute local path of a directory to be exported, followed by a space-seperated list of allowed clients. NFS can be extremely powerful, but I’m not going to go into the full details here. For our purposes, the following would do:

/etc/exports:

/vmail mx-1.internal.example.com(ro, insecure) mx-2.internal.example.com(ro, insecure) postman.internal.example.com(rw, no_root_squash) secure-mail.internal.example.com(rw, no_root_squash)

If you make changes to /etc/exports on a running NFS server, you can make the changes effective by issuing the command:

# exportfs -a

We’re going to need a real-user to handle all the virtual mappings in our setup. For this solution, we’re going to use user id 150. This user id is for a specifically created “Virtual Mail” user. It uses the standard “mail” group, with the default (Debian) gid of 8. You can create the user and directory like this:

# useradd -r -u 150 -g mail -d /var/vmail -s /sbin/nologin -c “Virtual Mailbox” vmail
# mkdir /vmail
# chmod 770 /vmail/
# chown vmail:mail /vmail/

NOTE: This user will need to be created on all servers which will be accessing this share (mx-1, mx-2, postman, files-1, secure-mail). If the user doesn’t exist, you could get file access errors. While NIS could be used to create a single host for this user, that’s outside the range of this document.

IV. Postfix on the Mail Exchange Servers

A. Re-building Postfix

The version of Postfix included in the Debian Etch distribution has broken quota support. We’re going to need to re-build it for quotas to work. This re-build process should NOT be done on the actual production MX servers. Have a seperate server sitting on the side (Or even just a seperate Virtual Machine in Xen) for performing build-tasks. Our server for this task is build.internal.example.com. On that server, install the needed build tools:

# apt-get install build-essential dpkg-dev fakeroot debhelper libgdbm-dev libldap2-dev libpcre3-dev libssl-dev libsasl2-dev postgresql-dev po-debconf dpatch libdb4.3-dev libmysqlclient15-dev lsb-release libcdb-dev

Then go ahead and download the source to the Postfix package:

# cd /usr/src
# apt-get source postfix

NOTE: Make sure you’re using the correct Postfix version with the following commands! This was written against Postfix 2.3.8. You can determine you’re current running postfix version by typing this at the prompt:

# postconf -d | grep mail_version

The output should look like:

# postconf -d | grep mail_version
mail_version = 2.3.8
milter_macro_v = $mail_name $mail_version

Assuming everything is good to go (and you’ve got the right versions), grab the quota patch:

# wget http://vda.sourceforge.net/VDA/postfix-2.3.8-vda.patch.gz

You then need to unzip the patch, and apply it to the new
source code:

# gunzip postfix-2.3.8-vda.patch.gz
# cd postfix-2.3.8
# patch -p1 < ../postfix-2.3.8-vda.patch

And then finally, rebuild the patched-package:

# dpkg-buildpackage

You may see a warning like this at the end of the dpkg-buildpackage command:

(Warning: Failed to sign .dsc and .changes file)

You can safely ignore this message.

This new rebuild will result in multiple packages being built, you want to copy postfix_2.3.8-1_i386.deb and postfix-mysql_2.3.8-2_i386.deb to your mail exchangers (MX-1 and MX-2). You are then ready to install!

B. Installing Postfix

Since we are using a Debian-based system, installing Postfix is a walk in the park. This subsection assumes you’re working on Mail Exchanger 1 (mx-1.example.com), if you’re working on MX-2, just exchange the server name in any config files, and you should be
good to go. Start the installation by running the following:

# dpkg -i postfix_2.3.8-2_i386.deb
# dpkg -i postfix-mysql_2.3.8-2_i386.deb

If/when the auto-configuration asks you questions about postfix during the installation, just select “No Configuration” (Since you’re using this guide, you’re going to configure it yourself!)

dpkg is going to install all of the configuration files for Postfix into /etc/postfix, so go there, and create the file main.cf:

# cd /etc/postfix
# touch main.cf

The main.cf file can be edited using two different methods. You can use your favorite text editor, or you can use the built-in postfix toolpostconf. We’ve already used postconf once to determine our version in subsection IV.A above.

The real benefit of the postconf tool is that it has some built in error checking, and it eliminates the possibility of ‘weirdness’ due to carriage returns, line feeds, odd quotes, etc. We’ll be using it in this guide, but there really is no requirement.

Start by filling in the basic information:

# postconf -e ‘myhostname = mx-1.example.com’
# postconf -e ‘smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)’
# postconf -e ‘biff = no’
# postconf -e ‘append_dot_mydomain = no’
# postconf -e ‘myorigin = example.com’
# postconf -e ‘mydestination = $myhostname, localhost, locahost.localdomain’
# postconf -e ‘mynetworks = 127.0.0.0/8’
# postconf -e ‘recipient_delimiter = +’
# postconf -e ‘inet_interfaces = all’
# postconf -e ‘proxy_read_maps = $local_recipient_maps $mydestination \
$virtual_alias_maps $virtual_alias_domains $virtual_mailbox_maps \
$virtual_mailbox_domains $relay_recipient_maps $relay_domains \
$canonical_maps $sender_canonical_maps $recipient_canonical_maps \
$relocated_maps $transport_maps $mynetworks $virtual_mailbox_limit_maps’

This is all you actually need to get a fully functional mail server running w/ standard unix users. We don’t want standard unix users, so we’ll be editing this file quite a bit. Remember though, change example.com to your own domain! (And mx-1 to mx-2 on the second server!)

B. Setting up Postfix for Virtual Users & Domains

Even though the mail exchangers won’t be delivering mail to the virtual users and domains, they will be rejecting based on valid/invalid destinations. We therefore need the mx servers to be able to connect to the SQL server to verify destinaions. Go ahead and put the information into postconf:

# postconf -e ‘virtual_alias_domains =’
# postconf -e ‘virtual_alias_maps = proxy:mysql:$config_directory/mysql_virtual_alias_maps.cf’
# postconf -e ‘virtual_mailbox_domains = proxy:mysql:$config_directory/mysql_virtual_domains_maps.cf’
# postconf -e ‘virtual_mailbox_maps = proxy:mysql:$config_directory/mysql_virtual_mailbox_maps.cf’
# postconf -e ‘virtual_mailbox_base = /vmail’
# postconf -e ‘virtual_minimum_uid = 150’
# postconf -e ‘virtual_uid_maps = static:150’
# postconf -e ‘virtual_gid_maps = static:8’
# postconf -e ‘virtual_create_maildirsize = yes’
# postconf -e ‘virtual_mailbox_extended = yes’
# postconf -e ‘virtual_mailbox_limit_maps = proxy:mysql:/etc/postfix/mysql_virtual_mailbox_limit_maps.cf’
# postconf -e ‘virtual_mailbox_limit_override = yes’
# postconf -e ‘virtual_maildir_limit_message = “The user you are trying to reach has exceeded their quota.”‘
# postconf -e ‘virtual_overquota_bounce = yes’
# postconf -e ‘transport_maps = proxy:mysql:/etc/postfix/mysql_virtual_transports.cf’

There are several ‘gotchas’ here. This configuration will host the virtual user mailboxes in /vmail. Should you need to store your mailboxes in another location, change the virtual_mailbox_base line accordingly.

The virtual_minimum_uid and virtual_uid_maps point to user id 150. This user id is for a specifically create “Virtual Mail” user. It uses the standard “mail” group, with the default (Debian) gid of 8. You can create the user and directory like this:

# useradd -r -u 150 -g mail -d /var/vmail -s /sbin/nologin -c “Virtual Mailbox” vmail
# mkdir /var/vmail
# chmod 770 /var/vmail/
# chown vmail:mail /var/vmail/

Now we need to install the NFS client tools:

# apt-get install nfs-common portmap

The directory then needs to be mounted to the NFS share. To do a simple test, run the following:

# mount files-1.internal.example.com:/vmail /vmail

You should now have access to the /vmail share from files-1. Try writing a file (It shouldn’t work!):

# cd /vmail
# touch tmp

NOTE: You should get a read-only error!

Assuming everything is working, go ahead and unmount the NFS:

# cd /
umount /vmail

And then make the mount permanent by putting the following into your /etc/fstab:

[...]
files-1.internal.example.com:/vmail /vmail nfs ro,rsize=4096,hard,intr,tcp,noatime,nodev,async 0 0

Go ahead and mount the file system one last time:

# mount /vmail

… and you’re good to go!

C. Postfix MySQL Configuration

Postfix was installed with MySQL support, but that doesn’t mean it already knows how to use our database. It needs to be provided with various SQL-query information for each type of table in our database. This information is stored in the MySQL files defined in the main.cf file. Note that in the following files, the last line contains a single comment (Preceeded by #) with the full query. Recent versions of Postfix can use this instead of the other statements. If you’re using a newer version, just comment out all of the other lines, and uncomment the query statement.

/etc/postfix/mysql_virtual_alias_maps.cf

user = vmail_user
password = vmail_user_password
hosts = sql-1.internal.example.com
dbname = virtual_mail
table = alias
select_field = goto
where_field = address
additional_conditions = and active = '1'
#query = SELECT goto FROM alias WHERE address='%s' AND active = '1'

/etc/postfix/mysql_virtual_domains_maps.cf

user = vmail_user
password = vmail_user_password
hosts = sql-1.internal.example.com
dbname = virtual_mail
table = domain
select_field = domain
where_field = domain
additional_conditions = and backupmx = ‘0? and active = ‘1?
#query = SELECT domain FROM domain WHERE domain=’%s’ AND backupmx = ‘0? AND active = ‘1?

/etc/postfix/mysql_virtual_mailbox_limit_maps.cf

user = vmail_user
password = vmail_user_password
hosts = sql-1.internal.example.com
dbname = virtual_mail
table = mailbox
select_field = quota
where_field = username
additional_conditions = and active = ‘1?
#query = SELECT quota FROM mailbox WHERE username=’%s’ AND active = ‘1?

/etc/postfix/mysql_virtual_mailbox_maps.cf

user = vmail_user
password = vmail_user_password
hosts = sql-1.internal.example.com
dbname = virtual_mail
table = mailbox
select_field = CONCAT(domain,’/',maildir)
where_field = username
additional_conditions = and active = ‘1?
#query = SELECT CONCAT(domain,’/',maildir) FROM mailbox WHERE username=’%s’ AND active = ‘1?

/etc/postfix/mysql_virtual_transports.cf

user = vmail_user
password = vmail_user_password
hosts = sql-1.internal.example.com
dbname = virtual_mail
table = domain
select_field = transport
where_field = domain
additional_conditions = and active = ‘1?
#query = SELECT transport FROM domain WHERE domain=’%s’ AND active = ‘1?

D. Client permissions

Since we’re looking to actually use this mail server in a production environment, we’re going to want to restrict who can send mail through it. Being an open relay (Accepting mail To/From everyone) is extremely dangerous, and a great way to get yourself labelled as a SPAM server.

Since the Mail Exchangers really only handle mail TO our domain, we’re going to be VERY restrictive. The next commands will tell the mail exchanger to allow email from the internal networks, and reject ANY email to an unauthorized destination.

# postconf -e ‘smtpd_recipient_restrictions = permit_mynetworks, reject_unauth_destination, permit’

E. Simple Spam prevention

While DSpam will be handling the ruling majority of our anti-spam measures, there are some simple tricks we can employ on the Mail Exchangers to take some of the load off.

1. Postgrey

Greylisting is a fairly effective countermeasure against spam, so we of course want to enable it on the Mail Exhangers. First of all, let’s get Postgrey installed. As always, Debian makes this simple for us:

# apt-get install postgrey

Postgrey will be injected before postfix sends the mail to the postman server, so we need to add it in the very end of the smtp_recipient_restrictions in main.cf.

# postconf -e ‘smtpd_recipient_restrictions = permit_mynetworks, reject_unauth_destination, check_policy_service inet:127.0.0.1:60000, permit’

Open up /etc/default/postgrey in your favorite editor, and change the options line to the following:

[...]
POSTGREY_OPTS="--inet=127.0.0.1:60000 --delay=55"
[...]

Then restart postgrey and incoming mail will be delayed by 55 seconds. You’d be surprised at how much spam this will prevent…

# invoke-rc.d postgrey restart

2. Postfix RBL and other rules

Postgrey is great, and will definitely trim down the amount of spam you see, but there are some more restrictions we can put into the exchangers before they go into production. I highly recommend checking out the postfix website and documentation to determine what all of these commands are:

# postconf -e ‘smtpd_recipient_restrictions = permit_mynetworks, reject_non_fqdn_hostname, reject_non_fqdn_sender, reject_non_fqdn_recipient, reject_unauth_destination, reject_unauth_pipelining, reject_invalid_hostname, reject_unknown_sender_domain, reject_rbl_client zen.spamhaus.org, reject_rbl_client list.dsbl.org, reject_rhsbl_sender dsn.fc-ignorant.org, check_policy_service inet:127.0.0.1:60000, permit’

It’s also worth it to restrict some other areas:

# postconf -e ‘smtpd_data_restrictions = reject_unauth_pipelining, reject_multi_recipient_bounce, permit’

F. Final Words on Mail Exchanger

So, at this point, your two mail exchanger servers (MX-1, MX-2) should be set up. These servers have RBL, DNS, and Postgrey SPAM prevention, as well as quota protection.

There is one item worth discussing here, and that is the transport table. Something to keep in mind is that this particular scenario was originally written with the intent of migrating from a qmail based delivery system to postfix. As such, we wanted to be able to control a slow migration, one domain at a time, without having to worry about DNS. We setup the two Mail Exchanger servers to handle all incoming mail, and originally, they just used the smtp transport agent to forward the mail to the qmail server (After checking postgrey). Then, using the SQL Transport queries, we were able to redirect one domain at a time over to the new format.

So, really, you could simplify the transport process if you’re starting from scratch. We weren’t, therefore we couldn’t.

V. Mail Delivery Server

The mail delivery server is the server that checks for SPAM using a bayesian algorithm via a service known as DSPAM. DSPAM is incredibly capable, and comes with a good deal of documentation regarding its various setups. Our particular setup is rather simple. MX-1 & MX-2 deliver mail to postman.internal.example.com via LMTP. LMTP is the Local Mail Transfer Protocol, you can find some basic information here. DSPAM receives the mail, checks against its own algorithms, and passes the mail over to ClamAV for virus detection. Assuming all is well, the mail is then given to Dovecot for final delivery.

We’ll go over the sections one at a time. So, let’s get things humming on postman.internal.example.com.

A. NFS Settings

Since Dovecot will be delivering our mail for us, we’re going to want to get the vmail directory mapped onto the mail delivery server.

Go ahead and create the vmail user and directory like this:

# useradd -r -u 150 -g mail -d /var/vmail -s /sbin/nologin -c “Virtual Mailbox” vmail
# mkdir /var/vmail
# chmod 770 /var/vmail/
# chown vmail:mail /var/vmail/

Now we need to install the NFS client tools:

# apt-get install nfs-common portmap

The directory then needs to be mounted to the NFS share. To do a simple test, run the following:

# mount files-1.internal.example.com:/vmail /vmail

You should now have access to the /vmail share from files-1. Try writing a file:

# cd /vmail
# touch tmp

It should work fine this time (postman has read-write access to the NFS mount…  the MX servers just had read access).

Assuming everything is working, go ahead and unmount the NFS:

# cd /
umount /vmail

And then make the mount permanent by putting the following into your /etc/fstab:

[...]
files-1.internal.example.com:/vmail /vmail nfs rw,rsize=4096,wsize=4096,hard,intr,tcp,noatime,nodev,async 0 0

Go ahead and mount the file system one last time:

# mount /vmail

… and you’re good to go!

B. Install DSPAM

Once again, debian makes this rediculously simple:

# apt-get install dspam dspam-doc libdspam7-drv-mysql

You will be asked if you want to configure the database for libdspam7-drv-mysql with dbconfig-common, choose ‘No’. Honestly, if you wanted to use dbconfig, there’s nothing stopping you, but for the sake of documentation, I’m going to spell out the settings

C. Setup the MySQL Database

You’re going to need to set up some tables in the MySQL database, so log in to sql-1.internal.example.com, and fire up the mysql client as root:

# mysql -u root -p

Then go ahead and create the new DSPAM database and user:

MYSQL> CREATE DATABASE dspam_db;
MYSQL> GRANT ALL PRIVILEGES ON dspam_db.* TO ‘dspam_user’@’postman.internal.example.com’ IDENTIFIED BY ‘dspam_user_password’;

NOTE: Change the dspam_user_password to a secure password!

D. Setup DSPAM for MySQL

Now that MySQL has a DSPAM database and User Account, we can go ahead and import the tables from the DSPAM MySQL driver. These are stored in /usr/share/doc/libdspam7-drv-mysql on the mail delivery server (postman.internal.example.com). There are two different versions of the schemas, one built for speed, the other to save hard disk space. We’ll be using the one for speed, but you can choose whichever you want. Remember, read the documentation, that’s why it’s there!

Install the main schema:

# mysql -u dspam_user -h sql-1.internal.example.com -p dspam_db < /usr/share/doc/libdspam7-drv-mysql/mysql_objects-speed.sql

If you get permission errors, then go back to the mysql server and check to make sure you made the user correctly.

Assuming all went well, we’re also going to want to feed in the tables for virtual users:

# mysql -u dspam_user -h sql-1.internal.example.com -p dspam_db < /usr/share/doc/libdspam7-drv-mysql/virtual_users.sql

Now, we just need to tell DSPAM how to talk to our MySQL DB. This is done via the /etc/dspam/dspam.d/mysql.conf file. Edit it accordingly:

[...]
MySQLServer sql-1.internal.example.com
MySQLPort 3306
MySQLUser dspam_user
MySQLPass dspam_user_password
MysqLDb dspam_db
[...]
MySQLVirtualTable dspam_virtual_uids
MySQLVirtualUIDField uid
MySQLVirtualUsernameField username
[...]

Now DSPAM will store all of its information inside of the MySQL DB.

NOTE: This is a temporary setup, just because we hadn’t finished the DSPAM virtual user install prior to writing this guide. Ideally, you’d want DSPAM looking at the same virtual user table as Postfix in order to get all the token information stored correctly. I’ll update the guide as soon as we’ve completed that change-over.

E. Install Dovecot

So we’re getting pretty far into our mail stack now… It has passed through the exchangers unharmed, DSPAM has gotten a hold of it, and now it needs to put it some where… but where? The answer is another service: Dovecot. Dovecot is an LDA (Local Delivery Agent), as well as a rather secure IMAP and POP3 server. As usual, Debian takes the majority of the work off our hands:

# apt-get install dovecot-imapd

For our purposes, we didn’t need POP3 (and really, neither should you), but if you wanted it, just append dovecot-pop3d to the end of that command.

And that’s it! Sure, there’s still some Dovecot setup to do, but not yet… we just wanted it installed so that DSPAM would have a place to put our email. Let’s go ahead and finish up the DSPAM install.

E. DSPAM Configuration

DSPAM is installed. DSPAM’s MySQL Libraries are installed. Dovecot is (very minimally) installed. What’s left? Well, while DSPAM is a truly incredible SPAM filter, it’s not so incredible that it completely sets itself up. (Although let’s be honest… give it a year, Debian will have it working 😉 In the mean time, let’s get this party started.

First, we need to make some changes in /etc/dspam/dspam.conf. This file contains all of the setup variables for DSPAM, and will tell it how to talk to the world.

/etc/dspam/dspam.conf

[...]
TrustedDeliveryAgent "/usr/libexec/dovecot/deliver -d %u"
[...]
UntrustedDeliveryAgent "/usr/libexec/dovecot/deliver -d %u"
[...]
QuarantineAgent "/usr/libexec/dovecot/deliver -d %u -m SPAM"
[...]
Debug *
[...]
Preference "spamAction=tag"
Preference "signatureLocation=message" # 'message' or 'headers'
Preference "showFactors=on"
Preference "spamSubject=[SPAM]"
[...]

You should also tell DSPAM about your local mail exchangers… this is dependent on your network setup, but for our scenario, MX-1.internal and MX-2.internal are 192.156.50.107 and 108, respectively.

[...]
LocalMX 192.168.50.107 192.168.50.108
[...]

Just a few more settings… remember, all of this can be found in the documentation, which you should read as often as possible!

[...]
Opt out
[...]
ParseToHeaders on
ChangeModeOnParse on
ChangeUserOnParse full
[...]

And finally, we need to tell the DSPAM daemon how to operate… we’ll tell it to wait for LMTP connections on port 2424 (The ‘unofficial’ LMTP port is 24, but since it’s unofficial, we’re going to use the >1024 insecure area).

[...]
ServerPort 2424
ServerPID /var/run/dspam.pid
[...]
ServerMode standard
[...]
ServerParameters "--deliver=innocent, spam"
[...]

One last little debian-related setup to do. DSPAM can be run as either a daemon (Always running service) or as a program call. By default, it’s setup as a program call, but we want the daemon to launch when the computer is booted. To change this, we need to change the value of START in /etc/default/dspam from “no” to “yes”:

# Variables for dpam.
#
# Do not start dspam.
#START=no
#
# Changed for DSPAM Daemon #
START=yes
# User that runs dspam.
USER=dspam
# Options for dspam
#OPTIONS="--debug"

That will set DSPAM to start automatically at boot. To continue our install, we’re going to start it manually:

# invoke-rc.d dspam start

And ~viola… DSPAM is now waiting for emails on port 2424 of postman.internal.example.com.

F. Training DSPAM

DSPAM can learn what spam is by reading your email, and over time, it will get better at figuring out spam vs. ham by your sending it “spam+user@example.com” and “nospam+user@example.com” messages. You can get a head start though, but giving it some initial training. Let’s go ahead and set that up.

You’re going to need bzip2

# apt-get install bzip2

Then you’re going to need to download the training files for DSPAM.

# cd /tmp
# wget http://spamassassin.apache.org/publiccorpus/20050311_spam_2.tar.bz2
# wget http://spamassassin.apache.org/publiccorpus/20030228_easy_ham_2.tar.bz2

Go ahead and decompress them:

# tar xvfj 20050311_spam_2.tar.bz2
# tar xvfj 20030228_easy_ham_2.tar.bz2

…and then finally, train DSPAM:

# dspam_train test spam_2/ easy_ham_2/

DSPAM will now process the files (This will take a while), and fill the database with the resulting tokens. When it is finished, it should be much more accurate.

G. Viruses!

But hold the phone! What if an email from a VALID source has a virus in it? Well, DSPAM has a nifty tag built into it for ClamAV. So let’s get that installed… Debian, again, is awesome:

# apt-get install clamav-daemon

Do a quicky edit of the clamav configuration file to uncomment the TCPSocket line:

/etc/clamav/clamd.conf

[...]
TCPSocket 3310
[...]

… and restart clamav:

# invoke-rc.d clamav-daemon restart

The edit /etc/dspam/dspam.conf

[...]
ClamAVPort 3310
ClamAVHost 127.0.0.1
ClamAVResponse reject
[...]

Restart dspam:

invoke-rc.d dspam restart

And they all lived happily ever after… DSPAM will now check for SPAM, and then check for viruses. SPAM will be put in the aptly-named “SPAM” folder, and the subject will be modified to start with “[SPAM].” Emails with viruses in them will get flat-out rejected by the server.

VI. Insecure IMAP

So now you’ve got a pretty darned bullet-proof mail server setup. Mail comes in, is run through a variety of anti-spam procedures, and is checked for viruses before being nicely deposited in a virtual user’s directory on the internal server. But… how do the users get to their email?

Good question. The answer varies depending on where the user is. For the time being, we’re going to assume that the user is either (A) On the local (internal) network, or (B) is accessing their email via a secure connection to the web server. For either one of these, they will need IMAP to be running. Since the postman server already has a fully-functional IMAP server installed on it (Dovecot), let’s use that server to do their accesses.

Open up the Dovecot configuration file located in /etc/dovecot/dovecot.conf and make it look the following. There is
probably already lots of default configuration done, so you will probably only need to uncomment certain sections and change minor things.

## Dovecot configuration file
#
base_dir = /var/run/dovecot/
#
# imap imaps pop3 pop3s (use imaps and pop3s if configured for SSL)
protocols = imap
#
# Uncomment the ssl_listen statements and comment out listen if using SSL
protocol imap {
listen = *:143
# ssl_listen = *:993
}
#protocol pop3 {
#listen = *:110
# ssl_listen = *:995
#}
#
log_timestamp = “%Y-%m-%d %H:%M:%S ”
syslog_facility = mail
#
# Uncomment these if using SSL
#ssl_cert_file = /etc/ssl/mycompany/mailserver/mail-cert.pem
#ssl_key_file = /etc/ssl/mycompany/mailserver/mail-key.pem
#ssl_ca_file = /etc/ssl/mycompany/ca/mycompany.pem
#ssl_verify_client_cert = yes
#ssl_parameters_regenerate = 168
#verbose_ssl = no
#
# Where the mailboxes are located
mail_location = maildir:/vmail/%d/%u
#
mail_extra_groups = mail
mail_debug = no
first_valid_uid = 150
last_valid_uid = 150
maildir_copy_with_hardlinks = yes
#
protocol imap {
login_executable = /usr/lib/dovecot/imap-login
mail_executable = /usr/lib/dovecot/imap
imap_max_line_length = 65536
}
#protocol pop3 {
#login_executable = /usr/lib/dovecot/pop3-login
#mail_executable = /usr/lib/dovecot/pop3
#pop3_uidl_format = %08Xu%08Xv
#}
protocol lda {
postmaster_address = postmaster@mydomain.com
sendmail_path = /usr/lib/sendmail
auth_socket_path = /var/run/dovecot/auth-master
}
#
auth_verbose = no
auth_debug = no
auth_debug_passwords = no
#
auth default {
 mechanisms = plain
 passdb sql {
 args = /etc/dovecot/dovecot-sql.conf
 }
 userdb sql {
 args = /etc/dovecot/dovecot-sql.conf
 }
 user = nobody
 socket listen {
 master {
 path = /var/run/dovecot/auth-master
 mode = 0660
 user = vmail
 group = mail
 }
 client {
 path = /var/spool/postfix/private/auth
 mode = 0660
 user = postfix
 group = postfix
 }
 }
 #
 # If you want client certificates, use these lines
 # ssl_require_client_cert = yes
 # ssl_username_from_cert = yes
}
NOTE: You can't just copy the above configuration and expect it to work! You're going to need to incorporate any changes above into your own configuration.

The Dovecot configuration is almost complete, but just as with the Postfix configuration, the MySQL settings need to be incorporated.

Open the file /etc/dovecot/dovecot-sql.conf and make sure that the following is present.

driver = mysql
connect = host=sql-1.internal.example.com dbname=virtual_mail user=vmail_user password=vmail_user_password
# The new name for MD5 is MD5-CRYPT so you might need to change this depending on version
default_pass_scheme = MD5
# Get the mailbox
user_query = SELECT '/vmail/%d/%n' AS home, 'maildir:/vmail/%d/%n' AS mail, 150 AS uid, 8 AS gid, CONCAT('dirsize:storage=', quota) AS quota FROM mailbox WHERE username = '%u' AND active = '1'
# Get the password
password_query = SELECT username AS user, password, '/vmail/%d/%n' AS userdb_home, 'maildir:/vmail/%d/%n' AS userdb_mail, 150 AS userdb_uid, 8 AS userdb_gid FROM mailbox WHERE username = '%u' AND active = '1'
# If using client certificates for authentication, comment the above and uncomment the following
#password_query = SELECT null AS password, '%u' AS user

Since we’ve stored our MySQL password in plain-text format in this document, we’re going to want to make sure we’re the only ones who can read it!

# chmod 600 /etc/dovecot/*.conf
# chown vmail /etc/dovecot/*.conf

The Internal/Insecure Dovecot configuration is now finished!  go ahead and resart the dovecot service:

# invoke-rc.d dovecot restart

… and your internal/webmail users can now check for email on postman.internal.example.com

VII. Secure Email

In an ideal world, our users would be able to send/receive email whenever they were on the net, from any place in the world. Unfortunately, that’s incredibly insecure… passwords are being tossed back and forth in plain text via the SMTP and IMAP protocols, and that means anyone who wanted to could just ‘snoop’ the password.

If your users don’t need direct access to the mail solution, then don’t give it to them! There’s no point in stressing over a secure email setup if all your users need is Webmail! Just make their connection to the webmail server secure, and make sure that the webmail server uses a secure network connection when talking to your mail servers. Problem solved! If, on the other hand, your users do require the ability to send/receive mail via the internet without using webmail, well then, that just makes things more difficult. Not impossible, just difficult.

So, here’s your problem: SMTP and IMAP send passwords over clear text. You can have them send passwords using MD5, but basic MD5 can be hacked. You can have them send passwords using MD5CRYPT, but then you’re dealing with multiple implementations (not to mention the fact that not all email clients support MD5 passwords). The solution? TLS (Transport Layer Security). We’re going to set up our solution to support an encrypted connection over the internet. While we could modify some of our existing servers to handle this, there’s no point in over-complicating their setups. We’re just going to run a seperate server to handle all of this: secure-mail.example.com

NOTE: In the original scenario, the small business had multiple static IP addresses. Since this is the case, we were able to run SMTP+TLS on port 25, if you don’t have multiple IP addresses, then that is not possible. The reason is simple enough: While IMAPS(secure IMAP) runs a different port (993) than standard IMAP (143), SMTP+TLS runs on the SAME port as SMTP (25). So, using a firewall to route based on ports allows you to run seperate IMAP and IMAPS servers, but no firewall in the world can route port 25 to two different machines. Even with all of that though, you could always just run SMTP+TLS on a non-standard port… heck, it would even be more secure.

So, with all of that in mind, we’re going to setup a secure mail server, which uses SMTP+TLS for sending mail, and IMAPS for receiving it.

A. SSL Certificates

The simplest form of encryption is having a simple self-signed certificate on the server. This will generate a warning message when the clients first connect, but they should be able to save it for further use. It is not really secure, since anyone can execute a man-in-the-middle attack if you don’t save the certificate.

The next level is using a server certificate signed by a Certificate Authority (CA), either a commercial one, or perhaps the company internal CA. This way, the server certificate will be trusted, and if you now receive a warning, there is potentially something bad going on.

Last but definitely not least is using client certificates for logging in to the server, and using a server certificate to authenticate the server to the clients. This is quite secure, but it is not supported in all mail clients. Thunderbird among others do have support for it.

1. Self-signed server certificate

First create the directories, create the private key, and lastly create the certificate.

# mkdir -p /etc/ssl/example.com/mailserver/
# cd /etc/ssl/example.com/mailserver/
# openssl genrsa 1024 > mail-key.pem
# chmod 400 mail-key.pem
# openssl req -new -x509 -nodes -sha1 -days 365 -key mail-key.pem > mail-cert.pem

Note that “Common Name (eg, YOUR name)” MUST match the name of the server, which in this case is secure-mail.example.com

2. CA-Signed Certificate

Using a real CA-signed certificate is no different from using a self-signed one. It’s just another step in the key-pair creation. If your company has its own CA, then they should issue a certificate for the mail server. A Google search for be your own ca will give you enough answers to create one yourself, if you have the need.

B. Secure Postfix+TLS

To begin with, we’re going to need to install postfix on secure-mail.example.com. This particular install doesn’t need quota support (It doesn’t handle local delivery), but just to keep things simple, we’re going to install it the same way we did above:

# dpkg -i postfix_2.3.8-2_i386.deb
# dpkg -i postfix-mysql_2.3.8-2_i386.deb

If/when the auto-configuration asks you questions about postfix during the installation, just select “No Configuration

dpkg is going to install all of the configuration files for Postfix into /etc/postfix, so go there, and create the file main.cf:

# cd /etc/postfix
# touch main.cf

The main.cf file can be edited using two different methods. You can use your favorite text editor, or you can use the built-in postfix toolpostconf. We’ve already used postconf once to determine our version in subsection IV.A above.

The real benefit of the postconf tool is that it has some built in error checking, and it eliminates the possibility of ‘weirdness’ due to carriage returns, line feeds, odd quotes, etc. We’ll be using it in this guide, but there really is no requirement.

Start by filling in the basic information:

# postconf -e ‘myhostname = secure-mail.example.com’
# postconf -e ‘smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)’
# postconf -e ‘biff = no’
# postconf -e ‘append_dot_mydomain = no’
# postconf -e ‘myorigin = example.com’
# postconf -e ‘inet_interfaces = all’
# postconf -e ‘local_recipient_maps =’
# postconf -e ‘local_transport = error:local mail delivery is disabled’
# postconf -e ‘smtpd_recipient_restrictions = permit_sasl_authenticated, reject’

You’ll notice that this time we disabled local delivery.  Since this is basically just an outbound relay server, we don’t want it trying to ‘deliver’
any mail… just send it forward.  We also set the SMTP server to only permit SASL authenticated sessions, and reject any other sessions.

Now we’ll want to fill in the information for SASL (SMTP Authentication). This does NOT encrypt the connection, it just requires the users to log in:

# postconf -e ‘smtpd_sasl_auth_enable = yes’
# postconf -e ‘smtpd_sasl_security_options = noanonymous’
# postconf -e ‘broken_sasl_auth_clients = yes’
# postconf -e ‘smtpd_sasl_type = dovecot’
# postconf -e ‘smtpd_sasl_path = private/auth’

So now your postfix install will query dovecot for all of it’s authentication needs, but it’s still not encrypted. Let’s go ahead and change that…

# postconf -e ‘smtpd_tls_cert_file = /etc/ssl/example.com/mailserver/mail-cert.pem’
# postconf -e ‘smtpd_tls_key_file = /etc/ssl/example.com/mailserver/mail-key.pem’
# postconf -e ‘smtpd_tls_session_cache_database = btree:/var/spool/postfix/smtpd_tls_session_cache’
# postconf -e ‘smtpd_tls_security_level = encrypt’
# postconf -e ‘smptd_tls_received_header = no’
# postconf -e ‘smtpd_tls_loglevel = 0’
# postconf -e ‘tls_random_source = dev:/dev/urandom’

Go ahead and reload postfix…

# postfix reload

And then let’s get Dovecot up and running…

C. Secure Dovecot

Dovecot server can do IMAPS pretty much right out of the box… so let’s get it installed on secure-mail. As usual, Debian takes the majority of the work off our hands:

# apt-get install dovecot-imapd

For our purposes, we didn’t need POP3 (and really, neither should you), but if you wanted it, just append dovecot-pop3d to the end of that command.

Open up the Dovecot configuration file located in /etc/dovecot/dovecot.conf and make it look the following. There is probably already lots of default configuration done, so you will probably only need to uncomment certain sections and change minor things. Pay close attention to the ssl_ sections of the file, as they are different on this server.

## Dovecot configuration file
#
base_dir = /var/run/dovecot/
#
# imap imaps pop3 pop3s (use imaps and pop3s if configured for SSL)
protocols = imaps
#
# Uncomment the ssl_listen statements and comment out listen if using SSL
protocol imap {
# listen = *:143
ssl_listen = *:993
}
#protocol pop3 {
#listen = *:110
# ssl_listen = *:995
#}
#
log_timestamp = “%Y-%m-%d %H:%M:%S ”
syslog_facility = mail
#
# Uncomment these if using SSL
ssl_cert_file = /etc/ssl/example.com/mailserver/mail-cert.pem
ssl_key_file = /etc/ssl/example.com/mailserver/mail-key.pem
#ssl_ca_file = /etc/ssl/mycompany/ca/mycompany.pem
#ssl_verify_client_cert = yes
ssl_parameters_regenerate = 168
verbose_ssl = no
#
# Where the mailboxes are located
mail_location = maildir:/vmail/%d/%u
#
mail_extra_groups = mail
mail_debug = no
first_valid_uid = 150
last_valid_uid = 150
maildir_copy_with_hardlinks = yes
#
protocol imap {
login_executable = /usr/lib/dovecot/imap-login
mail_executable = /usr/lib/dovecot/imap
imap_max_line_length = 65536
}
#protocol pop3 {
#login_executable = /usr/lib/dovecot/pop3-login
#mail_executable = /usr/lib/dovecot/pop3
#pop3_uidl_format = %08Xu%08Xv
#}
protocol lda {
postmaster_address = postmaster@mydomain.com
sendmail_path = /usr/lib/sendmail
auth_socket_path = /var/run/dovecot/auth-master
}
#
auth_verbose = no
auth_debug = no
auth_debug_passwords = no
#
auth default {
 mechanisms = plain
 passdb sql {
 args = /etc/dovecot/dovecot-sql.conf
 }
 userdb sql {
 args = /etc/dovecot/dovecot-sql.conf
 }
 user = nobody
 socket listen {
 master {
 path = /var/run/dovecot/auth-master
 mode = 0660
 user = vmail
 group = mail
 }
 client {
 path = /var/spool/postfix/private/auth
 mode = 0660
 user = postfix
 group = postfix
 }
 }
 #
 # If you want client certificates, use these lines
 # ssl_require_client_cert = yes
 # ssl_username_from_cert = yes
}
NOTE: You can't just copy the above configuration and expect it to work! You're going to need to incorporate any changes above into your own configuration.

The Dovecot configuration is almost complete, but just as with the prior configuration, the MySQL settings need to be incorporated.

Open the file /etc/dovecot/dovecot-sql.conf and make sure that the following is present.

driver = mysql
connect = host=sql-1.internal.example.com dbname=virtual_mail user=vmail_user password=vmail_user_password
# The new name for MD5 is MD5-CRYPT so you might need to change this depending on version
default_pass_scheme = MD5
# Get the mailbox
user_query = SELECT '/vmail/%d/%n' AS home, 'maildir:/vmail/%d/%n' AS mail, 150 AS uid, 8 AS gid, CONCAT('dirsize:storage=', quota) AS quota FROM mailbox WHERE username = '%u' AND active = '1'
# Get the password
password_query = SELECT username AS user, password, '/vmail/%d/%n' AS userdb_home, 'maildir:/vmail/%d/%n' AS userdb_mail, 150 AS userdb_uid, 8 AS userdb_gid FROM mailbox WHERE username = '%u' AND active = '1'
# If using client certificates for authentication, comment the above and uncomment the following
#password_query = SELECT null AS password, '%u' AS user

Since we've stored our MySQL password in plain-text format in this document, we're going to want to make sure we're the only ones who can read it!

# chmod 600 /etc/dovecot/*.conf
# chown vmail /etc/dovecot/*.conf

The External/Secure Dovecot configuration is now finished! go ahead and resart the dovecot service:

# invoke-rc.d dovecot restart

... and your external/insecure users can now check for email on secure-mail.example.com!

C. CA Signed client and server certificates

If you want to use CA-signed client certificates, you will need to take further steps, both in Postfix and in Dovecot to make this work. If you want the user names to be taken from the certificate itself, you currently must set the common name to the user name, for example user@example.com, which has been used in this document.

1. Telling Postfix about the Certificates

In Postfix, you can either use a directory of CA certificates, or a composite file with all the certificates concatenated together. We’re going to use the concatinated form, since that is what Dovecot is expecting.

# postconf -e ‘smtpd_tls_CAfile = /etc/ssl/example.com/ca/all.pem’

2. Telling Dovecot about the Certificates

In Dovecot, you must have the CRL together with the certificate for the authentication to work. The directives themselves are the following.

/etc/dovecot/dovecot.conf

[...]
ssl_ca_file = /etc/ssl/example.com/ca/all.pem
ssl_verify_client_cert = yes
ssl_require_client_cert = yes
ssl_username_from_cert = yes
[...]

NOTE: You will also need to change the password_query to the commented one in /etc/dovecot/dovecot-sql.conf

Warning: If you are running Dovecot release candidate 28 or older, the server will not send out the list of accepted CA names, which could make clients with multiple client certificates unable to connect. Please upgrade or install this patch.

3. Concatinating files

If you have several CAs and CRLs, it could be difficult to concatenate them each time, so a small script was created which will do that for you. Just stick it in your /etc/ssl/example.com/ca/ directory and run it. It will create an all.pem with all certificates and all CRLs.

make.sh:

#!/bin/bash
rm all.pem 2> /dev/null
cat *.pem *.crl > all.pem

4. Postfix TLS settings

Like I said before, there are some settings in Postfix that need to be changed as well, so let’s modify main.cf:

# postconf -e ‘smtpd_tls_ask_ccert = yes’
# postconf -e ‘smtpd_tls_req_ccert = no’
# postconf -e ‘smtpd_recipient_restrictions = permit_tls_all_clientcerts, reject’

Now you should have an enterprise ready email server with client certificates.

VIII. Final Words and Sources

So, assuming you’ve followed this guide through to its entirety, you should have one fully functional email solution. You’ve got a primary and secondary MX, a mail delivery server, a secure mail server, and a healthy dose of spam and anti-virus protection.

Does that mean you can just push the ‘GO’ button and walk away? Absolutely not. First off, you’re going to want to run the system through a number of tests… test everything. Try sending mail using your primary & secondary MX’s as your SMTP server (it shouldn’t work), try sending mail using your secure mail server, but without security settings enabled in your client (It also shouldn’t work). Try everything, and then when it’s all working, try it again. You don’t want to go rolling this setup into production without having tested it thoroughly first.

In addition, it’s entirely possible that something (or some things) are incorrect in this guide. A large portion of it was written from memory, and that always opens you up to a hole or two. If you find a glaring error, feel free to drop me a line, and I’ll get it fixed. This is a long guide, I wouldn’t be surprised if there were a few glitches in it.

Finally, I’d like to send out a ‘Thank you’ to the various people who helped me write this guide. There’s more than one source that I ‘borrowed’ from verbatim, and I couldn’t have written this without them. In no particular order:

HOWTO – Postfix and Dovecot with MySQL and TLS/SSL, Postgrey and DSPAM

Virtual Users and Domains with Postfix, Courier, and MySQL (Debian Etch)

Network File System (NFS) Server and Client Configuration in Debian

Postfix

DSPAM With Embedded ClamAV Integrated Into Postfix With Virtual Users And Domains



 

 

Comments

comments