Sanity check: Media Temple, SpamAssasin, IMAP, & the iPhone

  • November 9th, 2008

There is no denying that Gmail spam filtering is top notch. On average, I was fortunate enough to deal with less than 10 spam emails per week that managed to get through the Gmail spam filter. A free email service that offers accurate spam filtering? What’s not to like?

From my own experiences over the last two months, there isn’t a whole lot to like about intermittent account access – both via web or IMAP client. Previously, I preached the goodness of Gmail, often looking for ways to embrace and integrate the service into every day use.

Gmail is still a great email option over the likes of Hotmail or Yahoo. In reality, I have been making very little use of my actual Gmail email address for communication. I relied on the service as a central repository to index and manage multiple email accounts / aliases. I realized that if the "cloud" went down for maintenance or was disabled without warning, I was or would be SOL. Paranoia? Sure. But if you have the power to administrate your own services, why wouldn’t you? Email habits and requirements change. Mine have changed so much so that I’ve returned to self hosted email. Media Temple email + IMAP support.

Meet SpamAssassin, the frontline defense against internet scum

Back in August 2008, Media Temple deployed the Cloudmark Authority plugin for SpamAssassin.

The Cloudmark Authority plug-in for SpamAssassin increased Media Temple’s catch rate of spam, phishing and viruses by 80 percent and drove down false positives to almost immeasurable levels. Cloudmark’s high accuracy is due to a unique combination of Advanced Message FingerprintingTM technology and real-time corroborated feedback from Cloudmark’s Global Threat NetworkTM system, the world’s largest live threat monitoring system.

Back when I initially transitioned 5thirtyone.com from Dreamhost to Media Temple (2006), I tried the self hosted email route. It was as if the spam floodgates from hell had been opened. I had no choice but to rely on SpamSieve to clean-up where the server filtering fell short. Well short of expectations. Around that time, I had just begun hearing about Gmail and soon after adopted the service.

Fast forward to the present. Media Temple email is in top shape, IMAP support is working flawlessly, and all of my non-Gmail accounts reside on a server that I control. The improved SpamAssassin is working well enough for me to access my accounts via IMAP in Mail.app without the need for Junk Mail filtering or SpamSieve (server side filtering required! – see below). I may have just jinxed everything with that statement, but filters will continue to be trained to offset the increase of potential spam.

Server side filtering with procmail

A SpamAssassin installation that is properly identifying and tagging spam is of little help if tagged emails continue to land in your inbox. There are no automated filters out of the box with self hosted email. While you could setup a desktop filter to properly sort spam, what about mobile devices like the iPhone? One solution would be a desktop client running 24/7 applying rules on emails (IMAP). An obvious waste of resources. The second more logical solution is server side filtering.

Russ shares the two step process for automatically filtering spam tagged by SpamAssassin [server side] with procmail. Shared here for my own personal archiving. You’ll need access to your DV account via SSH. Enable developer tools from within your Media Temple account (if you have not done so already).

The first step requires the creation of a .procmailrc file within your email user account. I prefer vim over vi so:

vim /var/qmail/mailnames/your-domain.com/your-user/.procmailrc

Insert the following. Take note of the .Spam directory that I’ve specified (change if necessary).

MAILDIR=/var/qmail/mailnames/your-domain.com/your-user/Maildir
DEFAULT=${MAILDIR}/
SPAMDIR=${MAILDIR}/.Spam/
LOGFILE=${MAILDIR}/procmail.log
LOG="- Logging ${LOGFILE} for ${LOGNAME} "
# All mail tagged as spam (eg. with a score higher than the set threshold)
# is moved to the designated spam folder
:0
* ^X-Spam-Status: Yes.*
${SPAMDIR}

The second step requires editing the existing .qmail file for the user account:

vim /var/qmail/mailnames/your-domain.com/your-user/.qmail

The content of the file should look like this:

| /usr/local/psa/bin/psa-spamc accept
|preline /usr/bin/procmail -m -o .procmailrc

With server side filtering, you no longer have to worry about a barrage of junk mail making it to your inbox for manual sorting. A huge relief especially for mobile users.

The price of self hosted email vs. hosted services like Gmail

Let’s all be honest. The reason why Gmail spam filtering is so awesome is because of volume. The sheer volume of users continually teaching Google which emails are spam, and which emails are not. When you’re running your own email server, there is quite a bit more work involved to teach SpamAssassin – Spam vs. Ham.

Like any other spam filter, there is some training required. For each IMAP enabled account, I’ve created unique mailboxes / folders for training SpamAssassin. sa-learn comes in handy for training the Bayesian classifier. Media Temple uses the Maildir format (source).

sa-learn --no-sync [--spam or --ham] [folder/{cur,new}]

In my case since I’ve setup LearnSpam (for emails that slipped by SpamAssassin) and IsHam (for safe emails to learn):

sa-learn --no-sync --spam ~/Maildir/.LearnSpam/{cur,new}

and

sa-learn --no-sync --ham ~/Maildir/.IsHam/{cur,new}

Are you still using Gmail as your main email service? Or, are you utilizing your own hosted email, server side and desktop filtering?