By Joel Snyder and Network World Lab Alliance
Network World, 05/02/05
Original Artcle on Network World Web Site
A team of 20 iLabs engineers spent eight days running more than 1,100 SIP interoperability tests, and the conclusion is that having multi-vendor VoIP devices work together is by no means a given.
Last year , the VoIP using SIP iLabs team saw a high degree of basic interoperability between different SIP phones and proxies. When we expanded the testing this year to include enterprise VoIP features, standard security parameters and video (see story ), we encountered significant failure rates. Out of 1,113 interoperability tests 18% were outright failures.
All told, we tested 75 products from 25 vendors, including IP hard and softphones, videophones, SIP proxies, firewalls, and wired and wireless switches. Our test of enterprise features (see How we did it and test plan stories) focused on those required for large-scale VoIP deployments including call forwarding, call waiting, message waiting indicator, blind and attended call transfer, hold/resume with music, and dual-tone multifrequency testing. We chose features to test SIP interoperability, and didn't look at features that don't require interoperability such as last number redial.
We ran into some major interoperability problems early on in the testing. For example, 3Com's 655 Series SIP phones proved to be so recalcitrant in their SIP implementation that 3Com engineers pulled them from the test bed.
We locked up one of Grandstream's ATA-HandyTone 486 devices, and iptel.org's SIP Express Router running released software had serious problems completing calls properly that were only solved by upgrading the box to an unreleased version of the software.
While it's important to note that we had failures across the board, the enterprise feature that requires the most SIP interoperability work is call transfer. With call transfer, among the phones that supported both blind and attended call transfer, only 43% passed the test. Call transfer requires that many SIP messages get passed around to send the call between phones without dropping the call or the audio. In the case of an attended call transfer across a single SIP proxy, there is a minimum of 46 SIP messages that must align for the transfer to work properly.
Even with the less-than-optimal test results across all features, we had strangely anomalous behavior. For example, we conducted a significant amount of the testing via speakerphone. However, when we re-tested a Grandstream phone using the handset instead of the built-in speakerphone, it failed the same test it had previously passed.
Our tests show that, if enterprise features are required, picking phones and gateways from different vendors is a poor choice at this time. For example, we had difficulty in mixing and matching a Pingtel SIPxchange server with Zultys phones. Both were completely compliant with SIP standards, but Pingtel required a registration format that the Zultys phone couldn't be configured to send. Instead, network managers should focus on finding a single-vendor SIP proxy server and phone combinations, or a very small number of very well tested phone vendors that have a commitment to interoperability.
The SIP standards deal with very low-level operations. Because features such as "conference calling" and "attended call forward" are built by the phone vendors on top of the basic protocol, there are bound to be disagreements in how to use the basic protocol building blocks SIP offers to create these high-level features. Although there are some recent Internet drafts that help to map high-level features to SIP primitives, they have not yet been confirmed as standards.
SIP proxy vendors participating in this test say many of the features we tested eventually will be handled by the SIP proxies rather than the phones. This was evident when we looked at Asterisk - the popular open source PBX. The Asterisk code plays a very significant role in its implementation of the SIP call management protocol. By acting as a "back-to-back" user agent, Asterisk inserts itself in the middle of all calls. Therefore, interoperability between phones is not the issue because each phone only has to interoperate with the Asterisk proxy.
Phones attached to the Asterisk PBX had fewer failures than phones attached to the other SIP proxies.
SIP security success - or lack thereof
The primary intent of our SIP security testing was to find interoperable phone and SIP proxy implementations protecting both the SIP control channel,as well as the Real-time Protocol (RTP) datastream. Unfortunately, we not only didn't get to test these things - but we couldn't even find two phones that implemented the same security feature set. While many phones had individual security features, such as RTP datastream encryption, which worked within their own domains, we didn't find more than initial hints from vendors that encryption (from vendors such as Zultys and Cisco ) was going to be widely available.
We also found that some vendors aren't taking basic requirements such as authentication very seriously. As part of our test, we asked each SIP proxy vendor to hook up its proxy to NuFone, an Internet-based public switched telephone network provider, using SIP. To handle accounting, NuFone uses standards-based SIP authentication before allowing calls through its network. SIP authentication is a very simple operation. Thus, we were surprised that five of seven proxies could not perform authenticated SIP-backed communications.
To ascertain whether leading enterprise firewalls can handle SIP traffic, we set up an environment using the Asterisk PBX and four Cisco 7960 phones. Each phone was separated from every other phone with one of our test firewalls from Cisco, Check Point, and Juniper. We tested the fourth phone behind a home firewall, a Linksys network-address translation (NAT ) router, to simulate a normal broadband user environment.
We stacked the deck in favor of the firewalls by using Asterisk, which keeps both the SIP control channel and RTP datastream running down the same pipe. If we hadn't used Asterisk, we could have had a situation where the SIP control channel was going through one set of firewalls while the RTP datastream containing the actual voice traffic was using a different (but intersecting) set of firewalls. In this configuration, we called between each firewall-protected phone without problems.
In the case of the NAT-protected user, we reliably called out from the user through the NAT, but could not reliably call in. Because the Linksys router does not advertise that it is SIP-aware.
These results show that firewall vendors can cope with SIP traffic in this configuration. This is a step in the right direction because our testing last year with firewalls didn't achieve this level of interoperability.
In short, our testing suggests that enterprise customers looking to implement feature rich, secure VoIP deployments based on SIP, will have to wait for the products to evolve to that point.