Tests show a variety of top-quality gear available for enterprise users.
Original Article
from the Network World web site
While security vendors will sell you wares for any logical layer and all possible locations on your corporate network, IP Security-based VPN gateways have emerged as the most popular class of product for setting up secure, site-to-site connections.
In our evaluation of 13 products in this market, we looked at the standard manageability, performance and enterprise-focused feature criteria, but we also added a new series of interoperability tests to the mix.
In all, it was an astonishingly close race. No one product stood out as the winner across all categories. This give-and-take is reflected in our scorecard, where more than half the products are within a point of each other in the final tally. Due to the extremely close scoring, we will not be awarding a Blue Ribbon Award for this test.
Because of ambiguities in the IPSec standard specification, not every vendor's product will work with every other vendor's gear - even though each might have a "correct" protocol implementation. That makes interoperability one of the greatest challenges for VPN vendors.
We set up a hypothetical security policy for a large, multisite network and evaluated how well each VPN product could fit into that network. With multiple data centers and branch offices with switches, routers and firewalls, our test bed was designed to resemble a standard enterprise data services network. We tested interoperability of each product against every other VPN product, both in setting up initial secure connections and in maintaining long-term operation over a matter of days. Specifically, we rated how each product worked with the others, worked with our certificate authority and with popular VPN client software, and how well each handled different VPN authentication methods.
An important part of our evaluation was making products work in a full mesh. While almost every VPN vendor can do bake-off style interoperability (where they successfully negotiate a security association with one other vendor), we wanted to see what network professionals would be faced with trying to pin together a true multivendor network securely.
Our top interoperability category rating went to Secure Computing's Sidewinder. Right behind Sidewinder were products from Avaya, Check Point, Cisco, Microsoft, Nortel and Nokia. (Click here for an online chart showing the complete interoperability results.)
Although the objective scores based on our predefined evaluation criteria put Cisco almost at the top, we found severe restrictions in its implementation. Cisco has been both saddled and blessed with its command-line configuration interface. While some network managers love it, VPN designers will find it less than comforting. To keep VPN configuration manageable within the command line environment, Cisco allows for only a single Internet Key Exchange (IKE) policy (which can support multiple proposals and transforms) per system . While that worked fine for our test (we wanted a single corporate policy), it wouldn't work well in an environment where IKE policy varied across sites. In fact, when we added the SafeNet client into our interoperability test (see story, page 51), we were forced to break our Cisco PIX configuration: it could work with the policy we selected, or with the SafeNet client, but not both at the same time.
We used digital certificates issued by an Entrust public-key infrastructure (PKI) server for site-to-site authentication and discovered a huge variability in how the VPNs under test interacted with this PKI. Unauthorized access to company networks is a big problem, and one that VPNs can help to solve.
Sidewinder, Nortel's Contivity 1600 and Check Point's VPN-1 all provided decent evidence that we were in control of who was coming into our network and what resources they had access to. All three products let us specify in as little or as much detail exactly what kinds of certificates were needed to get into our network. In contrast, Lucent's Access Point 1000 and Cisco's PIX 525 and IOS (Cisco entered both IOS on an optimized 7140 VPN Router platform with hardware acceleration and PIX 525 as site-to-site products with different capabilities and design goals) had a more relaxed view of access control: Once you got a certificate from the PKI, you were free to make any kind of VPN connection you wanted. This could be a killer in an extranet environment, where not everyone is sharing the same PKI and where you got your digital certificate is often just as important as what the certificate says.
Hewlett-Packard's VPN Server Appliance, RapidStream's 2000, RedCreek's Ravlin 7160 and WatchGuard's FireBox III 4500 all failed our certificate interoperability test. In the case of RedCreek and WatchGuard, neither adequately supports external certificates for site-to-site authentication.
Microsoft's Windows 2000 Server with its built-in VPN service and Lucent's Access Point supported certificates, but had trouble talking to other certificate-supporting products. For example, with Lucent's Access Point, we had to drop back to preshared secrets (a more compatible but less desirable authentication system) when talking to Avaya and Nokia. Additionally, the Lucent product failed when talking to Cisco's PIX with our desired policy.
HP also had problems related to strange configuration requirements in its IPSec implementation. HP either wants to always initiate the IPSec security association, or never initiate the security association. In site-to-site networks, this is rarely the way things work. Thus, we had a hard time maintaining HP security associations, because of collisions when changing keys.
WatchGuard and RapidStream also had poor interoperability results. In WatchGuard's case, its unchangeable default security policy was insecure because it uses Data Encryption Standard (instead of Triple-DES) encryption and Diffie-Hellman Group 1, so we had to make a special case configuration in all other products to match the WatchGuard inflexibility. RapidStream 2000 gave us inconsistent results, sometimes accepting security associations and then failing to pass traffic, sometimes working perfectly.
As VPNs move out of pilot mode into implementation, configuration and management of dozens or even hundreds of VPN devices becomes a major issue. The only three products we tested that have GUIs fit for multisite, multivendor VPN management were Avaya, Check Point and Nokia. We took into account how these tools managed vendor-specific gear and how well each assisted in our interoperability scenarios.
Avaya and Nokia rose to the top of the enterprise configuration heap quickly. Both management tools use simple building blocks to define VPN network topology.
We created, modified and managed multivendor VPN configurations easily with these tools. Nokia's VPN Policy Manager GUI lets you build complex topologies: combinations of hub-and-spoke; mesh; links between the two; and individual tunnels between any two systems.
Avaya's GUI offers less topological power, but has a stronger back end. Built on top of a Lightweight Directory Access Protocol server, this product lets multiple network managers view, edit and push changes to a large network from multiple sites. Avaya's VPN Manager GUI also has a built-in client deployment and management tool for remote access VPNs. Nortel's Contivity 1600 has a similarly powerful tool built in, but the scope does not include more than one VPN security gateway.
We evaluated Check Point's recently released version of VPN-1 and were pleased to see that the company has come a long way in terms of VPN configurability and management since we looked at its V4.1 GUI earlier this year. Check Point's Policy Editor GUI can now handle a larger range of interoperability scenarios. Although we tracked down a few bugs in the new software, our general impression is that Check Point is focusing more on large VPNs. The Check Point GUI can easily generate meshed VPN topologies - a significant improvement from the V4.1 GUI, which tightly restricted a VPN manager's flexibility in selecting operating parameters.
Check Point's management tool also allows integration of firewall rules and VPN settings. Although Check Point's product does not support the comprehensive array of protocols and features that Lucent's Access Point does - which lets you stack firewalls and VPN tunnels in any configuration you'd ever desire - it handles 90% or more of the configurations most enterprise managers would dream up. The ability to express firewall filtering and VPN tunneling in the same rule is an essential element in merging VPN and firewall policy.
Several of the VPN management tools were not very useful in our interoperability quest. Nortel's famed Optivity tool is good for doing things to multiple Contivity boxes without having to touch each one, but doesn't really do anything for site-to-site VPN configuration, even where only Nortel systems are included. This is unfortunate because Nortel's element management (via a local Web server on the Contivity system) is the best system management tool of any product we evaluated. If Nortel could extend that management to multiple systems, it would have the knock- down winner in this category.
Lucent's optional QVPN Builder GUI went the distance in terms of managing a network of VPN devices as a single unit, but had a critical flaw: It was not designed to have anything but Lucent Access Points in the network. Because of this, we fell back to Lucent's command-line interface (CLI) for configuration. Unfortunately you can't use both the QVPN Builder and the CLI; you have to choose one. HP's management tool for its VPN Server Appliance has a similar restriction: It has some primitive capabilities to apply a single policy to multiple systems but does not have any way to integrate non-HP products into the configuration. WatchGuard's optional VPN Manager software leaves network managers in much the same boat. The lack of firewall integration of the VPN configuration also left us disappointed because WatchGuard's firewall configuration is so simple and intuitive.
Microsoft's VPN management is far from simple and intuitive.Although it is possible to create a single IPSec policy that could be applied to multiple systems - as long as they are all Win 2000 servers - the GUI is complex and confusing. After 92 screens, we still couldn't figure out whether it was better than the competition. Frankly, any security configuration tool this complex is asking for an error to happen, which is unacceptable in an enterprise network.
Cisco's new Cisco Secure Policy Manager will be a great boon to any network manager interested in using either Cisco PIX or its IOS-based systems as a firewall or VPN. With Cisco's CLI syntax slightly different from and incompatible with IOS to PIX, Cisco Secure Policy Manager wins major points for making it possible to design and synchronize firewall rules and intrusion-detection systems across multiple Cisco systems. In this case, though, "possible" doesn't mean "optimal."
Cisco Secure Policy Manager has dedicated tools for building VPN tunnels. Both mesh and hub-and-spoke topologies are supported. Unfortunately, there is no support for third-party VPN products - you have to fake out Cisco Secure Policy Manager by describing them as Cisco elements. More importantly, the VPN configuration is not well integrated with the firewall rules.
No VPN exists in a vacuum. VPN functions may need to be combined with other parts of the enterprise network. Common additions to VPN devices on the market include firewalls, high-availability features, routing protocol support, bandwidth management and quality of service (QoS), multiple interface options (besides Fast Ethernet, which is most common), and tunnel status monitoring and reporting. Technically, these are not VPN-specific features, but enterprise managers will find them useful adjuncts in their quest to build more powerful and capable networks.
One obvious winner in the "how many features can we add to a single box" game is Cisco. Its IOS-based product includes VPN capabilities as a sideline, with industry-leading routing, multiple interfaces, high availability, traffic shaping and firewall all built into the same system.
Microsoft's Win 2000 Server could also be considered the feature king - what other VPN device can also run PowerPoint and Flight Simulator? While more relevant features, such as simple routing protocols, traffic prioritization and basic firewall, are easily available, Win 2000 Server has an advantage when it comes to database operations for remote users. Because the Win 2000 VPN software is completely integrated with the Windows Active Directory authentication system, access control for remote users can be tightly controlled and managed from any Win 2000 system.
The obvious combination of VPN and firewall features makes collocation more the rule than the exception, with virtually every product we reviewed having at least limited firewall capabilities. The two holdouts are Nokia and RedCreek. (RedCreek announced an integrated firewall in its VPN product as this review was going to press.) We found Check Point and Secure Computing's approaches the easiest to deal with. Both offer an integrated firewall and VPN rule set. WatchGuard and HP are examples of the other extreme: The firewall and VPN are totally separate and disconnected services, which happen to share a GUI and sit on the same system.
Routing is another area where integration between firewall and VPN is important. Products like Nokia's CryptoCluster and Check Point's VPN-1 have no real internal connection between routing protocols and VPN status - tunnels come up, tunnels go down and the routing algorithm is none the wiser for it. Lucent's Access Point does an excellent job of integrating the two, representing tunnels as interfaces, which lets the routing system propagate information about the state of the VPN. This can be critical when a VPN tunnel is part of a back-up strategy or when multiple redundant paths exist across a VPN. In recent versions, Cisco's VPN products have added similar integration.
High-availability functions varied between products. Nokia, the unchallenged leader of the high-availability load sharing cluster, focuses on reliability at a single point in the network (see www.nwfusion.com, DocFinder: 6137). Other vendors, including Avaya and Nortel, built in reliability from a multisite or multilocation point of view.
Service-level monitoring and reporting can be important in secure enterprise networks. Lucent and Check Point have built-in tools to monitor the latency and loss rate of VPN tunnels and maintain an internal database of performance statistics that can be used for long-term charting or alerting purposes.
QoS is another enterprise-level feature that varies among products. For example, several of the VPN security gateways we evaluated have the ability to mark Differentiated Service bits on packets. Avaya's VSU series does this, but won't change its packet handling options based on QoS markings. This is primarily useful where other components in the network handle bandwidth management. Other products, including Nortel's Contivity and Cisco's IOS, mark and allocate bandwidth to tunnels based on configured-in rules.
We've found - both in this and in past tests - that performance of VPN devices varies widely. In many cases, vendors purposefully understate performance to drive sales to more expensive devices; in other cases, they overstate performance to make their products appear more competitive. While we did not conduct a comprehensive suite of performance tests - as that was not our primary objective for this review - we did take the opportunity to run some quick benchmarks to offer apples-to-apples comparisons of these products.
We ran three sets of performance numbers, evaluating behavior in best-case and worst-case packet flows, as well as with a typical Internet mix (see graphic, page 47). For the Internet mix, we used data collected from an Internet backbone to build a profile of approximately 50% small packets (96 octets or less), 10% large packets (1,518 octets, the Ethernet maximum transmission unit), 20% 576 octets (a common WAN MTU) and 20% assorted between 192 and 1,024 octets.
We discovered that for line speeds of up to 10M bit/sec (full duplex, about a quarter of a DS-3/T-3 circuit), any of the products can keep up - but Avaya, Nortel, RapidStream and Microsoft give you excellent price/performance ratios.
If you want to push to a full DS-3 circuit (45M bit/sec, full duplex), again using "real world" packet sizes, only Lucent's Access Point with dual cryptographic accelerators and the one-two punch of Win 2000 combined with Intel's Pro/100S cryptographic network interface cards (NIC) beat the 90M bit/sec needed to handle that circuit. By adding less than $200 worth of hardware to our system, we drove total IPSec performance of Win 2000 up to more than 160M bit/sec in the best case (large packets). Given the low cost of Pentium-based PCs, Win 2000 Server software and the Intel NICs, this particular packaging achieved price/performance ratios between 10 and 20 times better than the other vendors'. However, we note that our performance tests were done with only six IPSec security associations. As a central site system with 500 security associations, we saw total performance of our Win 2000 system drop dramatically to less than 8M bit/sec for the Internet mix.
Nokia offers a load-sharing product and we tested it in two ways: as a single stand-alone system (a single CryptoCluster 2500) and as a cluster (three clustered CryptoCluster 2500s). The results were fairly dramatic, showing almost linear growth in performance along with growth in cluster size.
When Cisco engineers originally configured their system on-site, they loaded an "E" series of software, which offers performance optimizations for its VPN. In our testing, we uncovered several interoperability problems with the "E" series software. For example, Cisco's IOS software wouldn't even talk to Cisco PIX VPN in some configurations. Cisco asked us to change to its "T" series software, which resolved all the interoperability problems. However, the "T" series software is not as performance-optimized as the "E" series and requires additional configuration to turn on certain performance features. The Cisco engineer on-site for our testing reviewed our configuration, but did not find the small details which turned on interface-specific performance optimization. When we re-ran the performance tests using the "T" series software the Cisco 7140 VPN Router platform did achieve better performance.
Enterprise network managers often need to mix everything from dial-up modems to 155M bit/sec OC-3 lines in the same network. For that reason, a one-size-fits-all approach won't work in the real world. This is one reason multivendor interoperability is so important: Small-office/home-office (SOHO)-sized products from vendors such as RedCreek and WatchGuard may be the right fit for some parts, perhaps talking back to gigabit behemoths like the Nokia 5205.
At the same time, a single vendor immensely simplifies management, as we saw so dramatically, and a broad product line can be an important advantage. Therefore we also looked at the breadth of VPN product line in our rating equation. We wanted to know how well vendors' product lines reached up to data-center-sized encryption engines and stretched down to the SOHO market with inexpensive simple devices. We also evaluated how well network media other than the vanilla Fast Ethernet were included: WAN interfaces such as T-1/T-3 and ISDN can be important in keeping costs down, while Gigabit Ethernet is critical for the high-end data center.
The obvious winner in this area is Cisco's IOS. With more than a dozen chassis options, products ranging from less than $1,000 to the Internet-core-sized GSR 12000, with its six-figure price tag, and interfaces ranging from built-in modem up to Gigabit Ethernet - no one beats Cisco's IOS when it comes to breadth of product line.
However, high scores in this category don't necessarily equate to the same level of flexibility, so investigate what hardware and speeds you need in this area carefully. For example, Cisco's PIX 525 rated well because of the large number of simultaneous interfaces supported, while Nokia's CryptoCluster 2500 got the same score because they stretch from branch office to data center in speed - although Nokia only supports two interfaces on its systems.
This review shows there is room in the VPN marketplace for more than a handful of vendors. Each product evaluated has specific strengths and weaknesses; each is designed with a different kind of network, management style, VPN size and set of requirements in mind.