New gateway products from MailFrontier and Cloudmark are effective but bypass some end-user control.
By Joel Snyder and Janet Trumbo
Network World, 02/24/03
Article on Network World Web Site
While desktop-based antispam software is widely used to keep unwanted, unsolicited mail in check, savvy corporate network managers now are pushing the onus of blocking spam out to their mail gateways. By blocking unwanted e-mail before it hits the corporate mail server, these products lighten the spam load on servers, system managers and end users.
We tested two products of this ilk that were introduced at IDG Executive Forums Demo last week in Scottsdale, Ariz. On the hot seat were Cloudmark's Authority and MailFrontier's Anti-Spam Gateway (ASG).
We conducted our tests at Opus One, a Network World Global Test Alliance member and e-mail and security consultancy, and found half the mail during our weeklong test period was spam (49.5%, to be precise). Both products can decrease the amount of spam substantially. Depending on your settings and product choice, between 80% and 90% of the spam coming into your corporate servers can be deflected.
However, based on our overall assessment of these products, they have a ways to go before they're ready for the typical enterprise deployment. Both take the decision of tuning what is and is not spam away from end users. This is a serious shortcoming because the inability to look through quarantined messages would be a major problem for any company that relies on e-mail for more than casual communications.
MailFrontier did an outstanding job of picking out spam - detecting 86% of the spam fired at it over seven days. But its dependence on Exchange and Outlook in this first version of the product and the requirement to add software to end users' systems, as well as some holes in its whitelist management strategy, counteract its superior spam identification algorithms.
On the other hand, Cloudmark's low-overhead, low-maintenance application looks more elegant, but has many of the same per-user customization problems as MailFrontier. Worse, of course, is the relatively spotty performance of Cloudmark's spam identification algorithm compared with the benchmark MailFrontier set.
Both companies have acknowledged they need to go further in letting users verify and control their spam, and plan to solve these problems in the next release of their products.
How they work
Cloudmark's Authority acts as a Simple Mail Transfer Protocol relay that is inserted in a message stream pretty much wherever you want, as long as it's before the messages hit the corporate mail server. Cloudmark delivered its relay to us as a plug-in to the widely used Sendmail mail gateway, which we ran on Linux 7.2 on a standard Intel platform. For testing purposes, we put Cloudmark Authority between our mail firewall and a mailbox server, but you wouldn't have to do anything that complicated. Because many companies use Sendmail as their mail firewall, you could simply add Cloudmark Authority to an existing Sendmail firewall.
Cloudmark Authority is simple once Sendmail is working. The configuration is stored in three text files that set the policy for handling spam and store the whitelist, a list of domains or IP addresses that will never be called out as spam. A blacklist - domains or IP ranges for which mail will not be accepted - is not explicitly supported in Cloudmark Authority, but is built into Sendmail.
As Cloudmark Authority peers at an incoming mail stream, it determines a score from 0 to 100 for each message, with the higher the score, the more likely it is spam. Depending on the score, it takes one of five actions: quarantine the message locally in a mailbox on the Cloudmark Authority server; drop and delete it entirely; return the message to the sender; tag the subject line (such as adding "[SPAM]") and send it along to the corporate mail server; or, add a header tag to the message (such as "X-Spam") and send it along.
Most network managers will block mail with a very high score (say, above 95), and tag mail that might be spam (with a moderate score, say between 70 and 95). Spam mail that is tagged, either in the subject line or as a separate header, usually can be placed in a separate folder by most clients to help divert spam out of the normal mail stream. Of course, any tagged mail still has to be downloaded by the user and eventually reviewed.
MailFrontier's ASG also is a SMTP relay and has a similar architecture, with a twist. We installed ASG on Windows 2000 Server, along with the included Web-based graphical user interface (GUI). The difference in architecture sits in two profilers included with ASG. The corporate profiler runs on a corporate mail server and watches the log files. As it sees users internally sending messages to addresses outside the company, it dynamically adds those addresses to its whitelist. This means that once mail is sent to someone, anything that person sends back will no longer be considered spam. The user profiler software, which must be pushed out to each client, then scans the address book and the sent messages and uses that to preload the whitelist by sending them to the ASG.
ASG's Web-based configuration GUI is easy to learn and use. In addition to normal management functions, it includes a small report writer for some basic statistics, whitelist/blacklist management tools and quarantine management. ASG has a fairly limited list of supported platforms: the corporate profiler supports only Microsoft Exchange, and the user profiler works only with full Outlook (not Outlook Express). MailFrontier officials say they will expand both in future releases.
ASG has three levels of spam identification: not junk mail, junk and maybe junk. Mail not marked as spam is sent to the corporate mail server without change. Junk and maybe junk either can be sent on untouched, deleted, quarantined on the ASG server, forwarded to a second address, or sent to the corporate mail server with the subject line tagged.
The design of both products could be a major problem in companies where end users demand to see their own quarantine files, or set their own spam thresholds and actions. ASG allows different users to have different actions. For example, some users could have spam filtering disabled if they wanted, but the network manager must set this parameter. With both products, the network manager must read through the quarantine files to identify false positives and help tune the whitelists. Because Cloudmark requires 1,000 users as a minimum and MailFrontier requires 2,000 minimum users, the quarantine files would become unmanageable within minutes of installation.
Both companies acknowledge that this is an issue and say that they are working on a way to solve these problems in future releases. MailFrontier has an additional software tool for Outlook Express and Outlook users called Matador that lets you manage your whitelist, but this adds yet another piece (and additional cost) to the deployment.
How well they perform
We tested both products by running a real-time stream of real mail messages through both products to see how they behaved (see How we did it). With 3,090 messages over a seven-day period, we got a good pile of both spam and nonspam to look at.
In the case of MailFrontier, performance was easy to gauge because there are fewer knobs to twist. MailFrontier did an excellent job both in identifying spam, reducing total spam by 86.1%, and in letting through good messages, with a false positive rate of 1%. When MailFrontier wasn't sure, marking a message as maybe junk, only 110 messages out of 3,090 fell into that category.
Cloudmark's 0-to-100 scale made gauging performance more difficult. We tried setting the thresholds in a variety of ways, and always ended either with an unacceptably high false positive rate or a very low spam-filtering rate. For the purpose of evaluating Cloudmark, we picked two thresholds between 0 and 100 and assigned the higher one the equivalent junk label, and the lower one the maybe junk level.
For example, if we set the junk category for Cloudmark to be a score greater than 98, and the maybe junk level to scores between 80 and 98, then the false positive rate drops to a very acceptable .6%, but the spam reduction stood at 43.7%, with the maybe junk category collecting 450 messages out of 3,090.
On the other hand, if we set Cloudmark's junk level to 80, with maybe junk between 50 and 80, then the false positive rate shoots up to an unacceptably high 5.3%, with a spam reduction of 62.9% and the maybe junk category collecting only 124 messages out of 3,090.
We also calculated false negatives: messages that are spam, but were not marked as such. Although everyone wants to reduce false negatives, some are inevitable in any system such as this. We thought that a false negative rate in the range of 10% to 20% would be acceptable, although the lower, the better. MailFrontier kept the false negatives to 4.2%, and Cloudmark's product had false negative reading of 16.5% to 18.1% depending on the settings we used.