Advanced Screening

We evaluated six leading firewalls to see how well they defend against application-layer attacks.

By Joel Snyder
Information Security, March 2004

Original Article on Information Security Web site

Andrew Bagrin has a lot of confidence in his firewalls.

"Our auditor was pen testing, looking for servers that were accessible from the outside," says Bagrin, director of business technology applications for Knoxville, Tenn.-based Regal Cinemas.

"They found one that looked pretty vulnerable, so we said, 'Why don't you try to exploit it? See if you can take it down.'"

The auditors threw everything they had at the server, but Bagrin's firewall -- Check Point's FW-1 NG with Application Intelligence -- blocked every attack.

Free Webcast: "Application-Layer Firewalling: Raise Your Perimeter IQ"

Join guest expert Joel Snyder as he separates the hype vs. reality of application-layer firewalling and digs even deeper into this month's "Advanced Screening" cover story. Log in to this live expert Webcast March 10 at 12:00 p.m. EST. If you miss it live, simply access the recorded archive all this month.

Log in to Information Security's sister site, SearchSecurity.com to view this Webcast.
Not long ago, the testers might have been able to overcome firewall protection and blow that server down. But new protocols, business requirements and threats from the outside and inside are driving firewall vendors to enhance their products with highly granular application-layer controls and intrusion prevention capabilities.

We looked at the newest releases from six firewall vendors -- Check Point Software Technologies, NetScreen Technologies1, CyberGuard, Symantec, Secure Computing and WatchGuard Technologies (see "About This Review") -- to gauge the capabilities of app-layer technologies in traditional perimeter devices.

Specifically, we evaluated these firewalls for their advanced application-layer controls, HTTP controls, intrusion prevention functionality, advanced management and overall functionality (see "Rating Advanced Firewalls").

Broadly speaking, the Check Point and NetScreen products are stateful packet-filtering firewalls. The other products we looked at have their most advanced security features in application-layer proxies -- although all have packet-filtering capabilities as well.

While granular control of security has always been a part of proxy-based firewalls, their stateful packet-filtering cousins are beginning to leap the barrier into deeper inspection of application-layer data and the greater control it offers. At the same time, we found that both types of firewalls have moved ahead by incorporating intrusion prevention technology.

Greater Depth

For many security managers, firewalls are there to support a simple "go/no-go" policy. Certain applications on certain servers -- such as FTP or mail or DNS -- are available on the Internet. Users on the inside are generally allowed to go out, and that's all. If that describes your policy, you'll be happy not only with today's firewalls, but the firewalls of 1995.

Chances are, though, your security policy demands a lot more control at the firewall. We investigated the ability of firewalls to enforce a much more granular policy. For example, most employees only need limited Internet access, and you don't want them to be able to push information out to the Internet either intentionally or unintentionally, perhaps because a virus or Trojan horse has taken over their computer. Or, you might want to prohibit people from clicking on .exe files in Web browsers. Perhaps malware such as Code Red is taxing your log files and Web servers, and you want to filter that traffic at the firewall.

All of these security policies require knowledge of the application protocol being used and precise controls.

Advanced granular policies seem like a perfect fit for proxy-based firewalls, and they are -- in theory. However, there's a big gap between what proxy-based firewalls could do and what they actually do. Firewall gurus, such as Fred Avolio and Marcus Ranum, have talked for years about the ability of proxies to look deep into application data streams, monitoring and filtering content, moderating access based on authentication and even translating data. Most commercial proxy firewalls only offer a subset of these possibilities, focusing on the most widely used protocols, such as HTTP, SMTP and FTP.

We found great disparities in the advanced features of the proxy-based firewalls. At the same time, packet-filtering firewalls have stepped up to add many of the same access control features, though they haven't come as far as their proxy-based counterparts (see "FW Wars").

Keeping HTTP Secure

HTTP features dominate firewalls because most enterprises let users browse the 'Net. It's no surprise that firewall vendors have focused much of their application-layer control here, since HTTP has an enormous number of security problems.

We identified several areas where firewalls incorporate additional control. As expected, support for these varied among products (see "HTTP Controls"). It's difficult to conclude that any one product is "better" than another, but Symantec, Check Point and Secure Computing generally showed the most robust control sets.

We expected that products claiming to be "advanced firewalls" would have some support for each of these HTTP controls:

• HTTP header filtering. Because of the large number of non-HTTP applications that run over the protocol, such as KaZaA file sharing and many IM apps, security managers may want to restrict or control access to these services, which can masquerade as normal Web applications. This capability lets the security manager differentiate between the applications allowed by security policy, such as Web browsing, and those that might have different rules, such as IM.
  
  
• File and MIME type, HTTP script and language blocking. Although virus scanners can identify malicious applications, blocking all files of a particular type, such as .exe files, or all files of a particular MIME type (e.g., "application/octet-stream") can help enforce security policy. For example, you can prohibit the downloading of programs or media. The same is true for scripts embedded in HTTP pages, such as ActiveX, JavaScript and VBScript, as well as Java applets.
  
  
• Virus scanning and URL filtering. Scanning for malicious code at the firewall in HTTP, FTP and mail traffic is a valuable component of a multitiered virus protection strategy. Similarly, some sort of URL filtering is valuable. This can range from simply blocking known malicious URLs, like common worms or attacks, to actively categorizing and blocking the porn-and-sports crowd.

There's a caveat here. HTTP-based applications are often run over encrypted HTTPS channels. While HTTP controls on firewalls can block access to particular sites, features such as download controls or virus scanners are generally ineffective with encrypted HTTP. Firewall vendors, concerned about security and performance, are loathe to break that encrypted channel.

Beyond HTTP

While proxy-based firewalls boast long laundry lists of supported applications, vendors have been less willing to provide explicit controls for each of these protocols. We differentiated "basic" from "advanced" based on the capability of the control to look into the content of the application.

TFTP is a good example. Most firewalls have a proxy or application-layer gateway for TFTP, since it's required by the protocol. In our evaluation, however, we were looking for advanced controls for things like file names for file transfer or whether users can upload or download data.

With firewalls deployed primarily as perimeter protection devices, the vendor focus on the most popular protocols may be appropriate. However, as security managers begin to push firewalls further into the enterprise network, it's more likely that they will begin to look to security devices for more than simple access control. Firewalls can become valuable adjuncts to host- or server-side security, enforcing granular access controls that we would normally expect at the server level. Previously, simply being on the corporate network was sufficient to give access to application servers. Now, as these servers become business-critical, and as Trojan horses and worms reduce confidence in internal systems, additional safeguards are welcome.

Regal Cinemas, for example, deploys internal firewalls throughout its far-flung system of 560 theaters, plus corporate offices.

"If traffic is going somewhere, it's through a firewall," says Bagrin. "Every theater, every manager could potentially access any system. We made sure no one could do that."

In our evaluation, we found little agreement on which protocols to protect, meaning you'll have to look closely at which firewalls fit your particular needs (see "Protocol Controls").

All of the firewalls we looked at added sophisticated controls for FTP and SMTP. Clearly, firewall vendors have had strong feedback on how little security managers trust their mail servers to sit unprotected against the Internet. These firewalls offer everything from virus scanning and antispam technology to attachment stripping, file-size limits and header rewriting.

CyberGuard had the only serious LDAP proxy, with the ability to intercept and block write operations while letting reads go through. Of course, most enterprises don't do LDAP through a firewall but, if you do, having the firewall back up the access controls and security on the LDAP server itself (and guard against stolen passwords) can add peace of mind.

Every vendor except Secure Computing ignored SNMP. Secure Computing had a proxy that could block read or write operations and filter so that only certain SNMP MIB variables are accessible. That's not as obscure as it sounds: Any company monitoring systems from outside its firewall has struggled with the question of getting the traffic safely through.

We were especially interested in VoIP. Enterprises looking to deploy IP telephony will need H.323 or SIP firewalls. Secure Computing, Symantec and Check Point all had built-in H.323 capabilities, with CyberGuard and Check Point offering SIP controls. However, the controls aren't particularly impressive. For example, Check Point had an extensive array of VoIP controls, but all were focused on restricting VoIP traffic to authorized systems. That's a good start, but not a very advanced proxy, probably because VoIP proxies or application-layer gateways are quite difficult to write. But we're raising the bar by looking for features, such as SIP proxy servers, built into the firewalls themselves. Secure Computing was unique in giving control over application behavior to block certain kinds of H.323 streams.

Some mainstream applications drew little or no attention. For example, anyone wanting to run Microsoft file sharing (CIFS) across a firewall should be looking hard at Symantec and Check Point products. They offer application-layer controls on CIFS, such as the ability to differentiate among operations (read versus write versus delete, for example) and limiting which shares can be connected to a particular server.

TFTP was completely ignored, even though it's commonly used in enterprise environments for software updating and configuration backups on network devices. The alternative is to open up TFTP generally for "trusted" systems and not use any application-layer access controls.

Scanning and Searching

Most of the firewalls feature virus scanning and policy-based access controls, with the exception of WatchGuard's Firebox X, which doesn't support virus scanning -- a startling omission. We found differences, though, in how each firewall implemented these functions.

Virus scanning can be a severe drain on resources. Since proxy-based firewalls tend to lag in performance, some vendors, like CyberGuard and Check Point, encourage (or require) virus scans to be on a separate server using the industry-standard CVP protocol. Nevertheless, having the option to have virus scanning run on the firewall itself seems especially valuable, especially in branch or remote offices, where a CVP server might not be available or where performance might not be an issue.

The same is true of policy-based access controls. Although the effort of looking up a URL to determine if it's "satanic/cult" or "gross depictions" requires fewer resources than scanning for viruses, it still presents an update and management problem. The firewalls we tested have some way of linking to a policy-based access control server like Websense, WebBlocker or Smartfilter. Because these URL lists often have millions of entries and a heavy update schedule, sysadmins traditionally have used external servers rather than locally storing the categorization information on the firewall.

However, some firewalls allow you to maintain a short block list on the firewall itself. With malware such as Code Red still filling up log files more than two years after it first appeared, it's simpler to block a few common malicious URLs at the firewall than let them get into the corporate network. Diligent sysadmins will also block (and alert) those URLs for outgoing traffic to help detect when their own systems have become infected.

Even if scanning for viruses is painfully slow, it's a bigger problem if you can't control where and when the scanning takes place. Most of the firewalls with virus scanning are limited to HTTP, SMTP and FTP. If you want to monitor other protocols, such as POP3 and IMAP, for virus content, you have to turn to NetScreen, the only product we found that lets you scan for viruses on any protocol and port you want.

Intrusion Prevention

The only agreement we found among firewall vendors regarding intrusion prevention is that it appears to be the hot topic for 2004. IPS features fell into two major categories: DoS prevention and anomaly/signature-based protection.

The goal of DoS prevention is to keep unusual volumes of traffic from overwhelming the firewall or the servers it protects; a single PC infected with a virus can bring even the largest enterprise firewall to its knees within minutes.

Anti-DoS protection in these firewalls, based on connection rate limiting, is fairly primitive compared to dedicated IPS products from vendors such as Top Layer, Captus Networks and DeepNines Technologies, which also look at bandwidth consumed by applications, sources and destinations, as well as historical trends in bandwidth usage for particular servers.

The general term used by firewall vendors for connection rate limiting is "SYN flood protection," although ICMP and UDP connections can be just as important to good DoS protection.

SYN flood protection can take many forms. On one end of the spectrum, WatchGuard has just a single set of SYN flood parameters for the entire firewall that are focused on detecting unacknowledged connections. In contrast, NetScreen protects both the firewall and the servers behind it and has a series of thresholds for TCP, UDP and ICMP connections, as well as internal thresholds for the firewall's own connection table. As these are tripped, the firewall begins to more aggressively age connections out of tables to keep the packets flowing.

Secure Computing, WatchGuard and Check Point all combine DoS protection with packet blocking (sometimes called "shunning"). The security manager has the option to drop all traffic from a particular source that has been determined to be sufficiently annoying. However, this technique has its dangers, as spoofed packets can falsely trigger DoS protection, creating a self-imposed denial of service by blocking traffic from the legitimate system being spoofed.

NetScreen, Check Point, Symantec and WatchGuard all feature both anomaly- and signature-based protection. While innovative, their technology is a far cry from that of dedicated IPS devices from vendors like TippingPoint and Internet Security Systems. High-end, dedicated inline IPS devices include many more options for blocking traffic in response to problems, such as varying the behavior based on the particular signature or anomaly triggered; and more ways to modify the connection as it passes through the firewall, such as sending TCP RSET packets in one direction or the other based on the type of attack.

WatchGuard has the most basic IPS implementation, with a few anomaly-based rules (such as TCP packets with illegal flag options), but no IDS-based attack signatures. Instead, WatchGuard relies on anomalous activity to trigger evasion and intrusion prevention capabilities. For example, any proxy can be set to "autoblock," meaning that further traffic from a particular system can be dropped for a specified period of time -- even if it's legitimate.

NetScreen, Check Point and Symantec feature more robust IPS. Check Point's signature database is quite small, including protections for only six protocols in its SmartDefense technology (FTP, HTTP, SMTP, CIFS, DNS and SIP). Also, SmartDefense only looks at a bare minimum of protocol anomalies, including packet lengths, options and flags, as well as SYN flooding and a few signatures.

NetScreen extends coverage to IMAP and POP3, and has a much more extensive set of options for evasive action than Check Point, such as dropping the packet or closing the connection when a signature is triggered.

In contrast, Symantec incorporates an entire IDS with about 750 different signatures. Approximately 30 of these signatures are set to block traffic. The signatures are those representing the most critical threats and/or the least likely to set off a false alarm. This is customizable -- you can enable IPS for any IDS signature.

CyberGuard offers the option of mirroring traffic to a NetProwler IDS sensor. If the IDS decides that traffic is malicious, it can direct the CyberGuard firewall to shut down the offending connection.

Mixed Blessing

High levels of control can be a curse as much as a blessing, especially with today's configuration GUIs. Because most firewall GUIs are based on tables of rules that allow sources, destinations and services, throwing any other variable into the picture -- such as user authentication or URL regular expressions -- multiplies the rules and makes them unwieldy.

A security policy that varies strongly by user group, for example, will be very difficult to express in some of the GUIs we looked at, and impossible in others. There's a heavy administrative penalty for attempting to be too granular. For example, WatchGuard's proxy configuration is typical of proxy firewalls, with settings contained within the proxy itself rather than in a separate rule base. Since the proxy for a particular protocol holds all the settings, any variation means you have to create a second proxy. Thus, if you wanted to handle firewalling of external mail servers differently from internal ones, you'd have to create two SMTP proxies -- or three, or four, depending on your requirements.

For small networks, this isn't a big deal, but as things get complicated, the GUI gets much more complex quickly. In contrast, Check Point's configuration is an integral part of the rule base, which takes granularity down to the URL level. Most security managers will be happy with either configuration model because most security policies don't require a high level of granularity; but, if you do, investigating whether the firewall's GUI will easily support the granularity you need is an important part of your evaluation.

New Directions

As the Internet's threat model has evolved, firewall developers have made an effort to match the attacks with defenses. With greater application-layer awareness, both proxy-based and stateful packet-filtering firewalls give the security manager more granular controls and more precise adjustments on traffic flowing through the firewall.

While there's considerable commonality in some basic features, such as HTTP protection and SYN flood protection, these products are going in very different directions. For example, after a decade of competition, CyberGuard and Secure Computing, two of the oldest proxy-based firewalls, still don't cover the same set of application-layer proxies. This means that it's going to be harder than ever to find the right firewall for complex enterprise environments. While the spate of application-specific firewalls can help, security managers looking to reduce the number of boxes and systems they manage will need to go through a careful evaluation process to be sure the features they need are there.

Incorporating IPS technology into the firewall is clearly a step forward, but these capabilities don't approach the power and flexibility of stand-alone IPS products. Security managers with more precise DoS and/or anomaly/signature-based IPS requirements will need dedicated hardware, at least for now.

We found that most firewall vendors have been designing the advanced features of their products in a reactionary mode. Application-control features are largely designed to combat known threats rather than give the security manager control over content and permitted operations. Vendors are more likely to add signatures and application controls to meet demand. This is much easier -- and less expensive -- than building more sophisticated firewalls with a lot of granular and flexible control options. Since the proxy-based firewalls we looked at are all mature products with eight or more years behind them, it's unlikely that this trend is going to change quickly. Because of this, firewall vendors have already lost ground to newer kids on the block, such as SSL VPN devices.