IDS finds niche as analytical tools

By Joel Snyder, David Newman and Rodney Thayer, Network World Global Test Alliance
Network World, 10/13/03

Original Article on Network World Web Site

Network intrusion detection systems can be highly useful additions to your enterprise security arsenal. They provide unique visibility into your networks and offer powerful forensics tools that help detect how and when your network was attacked.

IDSs are not for every network, but when they are deployed in the right place, at the right time and monitored by the right network security professional, it's the right kind of product.

Those are the conclusions we reached based on our tests of five IDS products handling live Internet traffic for 60 days in real-world scenarios.

We tested these products as we did last year in front of multiple live networks that were open to Internet attacks. While last year's testing centered on simple detection, we went beyond that this year to focus on specific scenarios that an enterprise security manager would encounter.

We found that while false positives are still a problem, they are much less of a problem than they were last year as the vendors have gotten much better at managing the flood of false alarms.

While we invited more than a dozen vendors to participate, only Barbedwire Technologies, Cisco, Internet Security Systems (ISS), Intrusion and NFR Security took part in the end. (See "Equipped to play" for detailed description of the hardware and software each vendor brought to the test.)

Overall, we found that different products have different strengths, depending on your needs, such as:

• ISS has the most powerful management and analysis tool kit.

• Cisco provides a great deal of flexibility with its sensors and tight integration with its routers and switches. But its overall management lags the competition.

• NFR is best if you're going to be writing a lot of your own attack signatures.

• Intrusion's product set is nearly as strong as ISS, but with considerable rough edges and some notable gaps.

• We were less enthusiastic about Barbedwire's appliance. Its IDS implementation does not meet the needs of the enterprise network.

Scenario 1: What happened to Paul?

Before the big viruses hit in August, one of our IDS-protected Windows 2000 systems - named Paul - was cracked and being used as a zombie to scan other systems. We wanted to know who broke in and how.

Most products distinguished between alerting and forensics. In alerting, IDSs bring recent high-priority events to your attention. In forensics, they let you dig down to find the source of the problem.

Some products were very modal: You're either working in alert mode or in forensics mode, and there's a barrier between them. Intrusion, NFR and Cisco (with its Cisco Threat Response [CTR] alerting console it picked up through the acquisition of Psionics earlier this year) fell into this category.

ISS and Cisco's original IDS Management Console didn't differentiate between the two types of analysis. Barbedwire also takes a combined approach, mixing forensics and alerting into one interface.

We figured that because Paul got hit on a Friday, there should be a nice, juicy alert sitting there when we logged on Monday.

Intrusion's team had tuned our system to dump alerts after three days, so there was nothing to be seen. We adjusted the thresholds and discovered a nice feature: High-priority alerts can age differently than low-priority alerts. Not a massive competitive advantage, but a good sign that product developers thought about this. When we switched over to Intrusion's Forensics view, a handful of bugs in this early release prevented us from getting a good look at what that feature has to offer.

ISS impresses

Then we turned to ISS. ISS doesn't distinguish between alerts and forensics, but instead offers different views of the same data within SiteProtector, ISS' tool for managing and analyzing information collected from its security tools suite. One powerful feature is its automatic summarization function. Events are grouped wherever possible to reduce the report size. It was easy to confirm that Paul was hacked and when it happened.

To go from the "when did Paul get cracked" screen to "how did it happen," copy the IP address with a right-click, paste it into the same screen, and select a different view of the data. Wait 8 seconds, and there's your answer. Seems simple, but it was a sharp contrast to other products that don't have as sophisticated an interface for slicing, dicing, sorting and finding data.

SiteProtector now had a short list of events, of which six were listed as high priority. Powerful forensics capabilities let you identify attackers and see what else they might have been doing. More importantly, ISS' event management tool lets network managers dump the most interesting events into an "incident" folder and quickly generate a short list of research action items for any node. The linkage between events and ISS' X-Force database, with exact links to explanations and patch locations, gave us a huge head start on tracking down the problem.

NFR comes up short

The NFR interface makes a huge distinction between alerting information and forensics. In the product's attack signatures, the signature chooses when to send an alert, to actually record data to the database side, or when both are appropriate. Starting with the alerting side, we sorted by source IP address and discovered ... nothing. By default, NFR only displays 1,000 alerts. We adjusted that number up and discovered when it is set to 10,000 items, it takes 85 seconds to repaint the screen. Make it 20,000 items, and you'll wait nearly 5 minutes for a refresh. After we finally verified that NFR saw Paul attacking the world, we turned to the hard question of how Paul got hacked.

The short answer is that you can't ask the NFR product. When filtering alerts, you can't ask the NFR product "show me all the alerts with a destination of this IP address." You could get all the alerts for a particular time period and sort them by destination IP address, but you can't trim the list of alerts down to just the set you want. That was a little disconcerting, but we thought instead that we'd find the answer in the forensics side of the product. Unfortunately, you can't do that on the forensics side either.

We ended with a huge pile of irrelevant alerts and had to paw through them manually, scratch pad and Web browser at the ready, to figure out what had happened to Paul. In this case, though, NFR also missed the attack.

Cisco: Better data, bigger headache

With Cisco's CTR, you're given views by event, source and target, but you don't get the ability to easily jump around. You can discover the start time for attacks and then see who was attacking. But doing so requires a lot more GUI navigation than with other products. You can't trim your view by time (although you can sort by time), and you can't quickly match up source and target because the pre-programmed views don't allow for that. Because CTR only shows alert data, you only have the most recent information at hand, which means that when you want to run forensics queries, you have to jump over into Cisco's other tool, the IDS Monitoring Center, which is shipped as part of its larger management platform, CiscoWorks. Because the IDS Monitoring Center predates CTR, it also can act as an all-in-one interface, both for handling alerts and for conducting forensics research.

The bottom line is that what was fast using ISS' interface took significantly longer using either or both the tools Cisco provided. What came out, though, was different because CTR downgraded some of the alerts that ISS thought we should be concerned about. CTR did this when it saw that the attack was not successful. Our shortlist of activities was shorter with Cisco. The slightly better data did not offset the frustration of jumping between GUIs, using scratch paper and having to scramble around because there was no incident management system.

Minesweeper bombs

Regarding Barbedwire's Minesweeper, it's clear that coming back to this system three days after the attack was a major mistake. The attack wasn't in any of the alert reports that come back quickly from Barbedwire (as tuned by Barbedwire engineers) because they don't store the reports for that long. Even so, the reports wouldn't have helped because we don't want just the most recent alerts, we want to trim alerts by time, attacker and attackee. Whether this product can perform such a task is immaterial because it doesn't respond fast enough. After 30 minutes of not being able to complete a report, we moved on.

Scenario 2: How about those alerts?

Not wanting to repeat our experience last year with false alerts, we turned to the question of tuning. How easily and quickly could we trim the alert list of noise? Security managers come at this problem from two tacks. Some want to see all their alerts, but have them prioritized. Others simply want to throw out useless information.

Because we had done a lot of research on Paul as part of our first scenario, we created a simple task: make sure that Paul's false alerts would never be seen again.

Partial success with ISS

With ISS, we found two easy techniques for filtering events. Within the per-sensor policy, a well-hidden tab lets you drop events from ever being sent to the IDS console. Whether this would work for a large network is a tough question. Each combination of event and destination address has to be entered separately, which means that if you manage hundreds of servers, you could spend a long time dropping events. Or, more typically, you'd decide that the event wasn't important and disable it. But that's a dangerous approach because new systems popping up on a network might not have all the right patches applied and configuration changes made. Solving this problem the ISS way would be extremely tedious for large networks.

For security analysts who want the data, but don't want to see it unless they ask for it, ISS has a partial answer. For any view of the event and forensics data, a filter can be added that blocks particular attackers, target IP addresses and events from appearing.

Cisco's answer is complex

Cisco's IDS Management Console offers the capability to filter out particular signatures but in a complex way. We credit Cisco's IDS distributed management because it lets you put sensors into groups and then apply settings and configuration changes at the sensor level, group level and network level. At first glance, this is a mature and reasonable way to handle a network of sensors. But Cisco doesn't let you manage signatures and filters at the group or network level. So if you want to build a filter to drop out certain events, you have to go from sensor to sensor, copying the same information.

If you actually want the data, but want to separate the useless from the useful, Cisco offers a partial interface with its CTR tool. CTR provides a series of policies that are used to upgrade and downgrade alerts. But creating a new policy to say "this alert is not relevant" is difficult.

Barbedwire steps up, sort of

Barbedwire solved this problem by dropping events at the IDS level. To filter out a particular event, Barbedwire lets you give a list of IP addresses to which an event does not apply. It's somewhat tedious to do that, but conceptually it's an easy extension.

The only problem is that, like Cisco, there is no way to do this for more than one system at a time. If you have multiple sensors, you have to create the drop filters on each one. With Barbedwire's multi-port sensor, you'd have to do the same set of filters on each interface. Overall, Barbedwire's multi-system management was the weakest of the products we looked at. It had no ability to group or aggregate devices or even interfaces through the GUI, and there was effectively no central management, only central logging and reporting.

Barbedwire had no way to filter out events once they were logged; if you saw them once, you'd see them forever.

NFR has solid event filtering

NFR's event filtering worked at both the sensor level and at the GUI level. Sensor-level filters let you individually block data from the alerting or forensics part of NFR. Simply pick an event and define filters for it, applying it to individual sensors or to all sensors at once. It only took a single click to push changes to NFR sensors.

However, NFR fell down in the area of alert filtering. We spent a lot of time puzzling over GUIs and documentation before coming to the conclusion that dropping alerts in the NFR GUI is a waste of effort. But because NFR handles alerts separately from forensics, there is little reason to drop them in the alert GUI. If you're going to bother to filter out things, it makes more sense to do it in at the event level, where you have more control.

Intrusion offers efficient sensor tuning

Intrusion's tuning facilities vary depending on where you want to filter. In our case, trimming at the sensor was efficient, so we used that method. Intrusion's Policy Editor runs on the central management console and lets you build a policy that drops IP addresses from events as appropriate. From there, after a bit of technical support, we pushed changes to the sensors and trimmed the alert load considerably. Intrusion has made policy management of its sensors easier than it was last year, but it's still a lot harder than it needs to be.

Intrusion also supports pure IP filtering, but this requires direct access to the sensor via its Web interface and is not managed centrally. It sounds like an obscure feature, but the ability to block entire ranges would be important in a large enterprise deployment where multiple sensors saw intersecting traffic loads.

Scenario 3: Writing our own alerts

Not every network manager will want to write his own alerts, but we had a specific problem to solve. One worm that struck during our test was going to send traffic to some known hosts out on the Internet. We wanted to catch this traffic, put an alert on it and use that to help disinfect our network.

Cisco has the most obfuscated method for editing and modifying signatures. Rather than present signatures to the network manager in a simple text file, it gives a complex array of engines and parameters. Wizards help shield the manager from knowing all the details, which means we easily could generate the signature we sought. However, Cisco's tools for building signatures only work well for certain common cases, and we could see that there were some kinds of signatures that we weren't going to be able to build.

Barbedwire uses Snort as its underlying engine. Snort's signatures are easy to read and write, and coming up with a signature for Barbedwire was simple.

NFR provides a powerful development environment for its proprietary N-Code language. Clearly, any security manager who wants to write a lot of signatures will gravitate toward NFR's tool kit. NFR has an array of policy-based signatures that trap traffic based on policy rather than attack detection. We used an out-of-the box policy-based signature first and then tried writing our own N-Code program. N-Code is powerful, but the existing policy-based signature was a lot easier.

Accomplishing the task at hand with Intrusion and ISS was a bit frustrating because the core signatures and technologies inside of their products are hidden. ISS tries to solve this problem by offering the Trons language, which lets you put Snort-syntax signatures into the existing rule base. Intrusion's Policy Editor has a limited GUI that lets you create new signatures. Our requirements were simple enough that we generated the needed signature without going beyond the provided interfaces.

Fourth scenario: Finding the worms

Because our test network was intentionally behind on its patches, we knew the Microsoft RPC DCOM vulnerability would hit us hard. The question was who got hit? In a large network, being able to ask that question and quickly find (and patch) those machines would be a high-priority security problem to solve. These systems are especially easy to find, because they start scanning other networks, looking for systems to infect. We turned to the forensics features of the IDSs to help.

Barbedwire was a disappointment. The issues we had with performance came back to haunt us. Because the infected systems were generating traffic at a furious rate, we had databases with more than 100,000 events each day. That's not an unreasonable amount for an enterprise network to generate on a bad day, but it was way too much for the Barbedwire systems to handle. Any attempt to generate reports just didn't work.

With NFR, the key to any forensics investigation is figuring out what event you're looking for. We knew from CERT's advisory to look for PING traffic and dove into NFR. Our first guess, "ICMP Pingflood," turned out to be wrong, but after a few seconds, we came up with "IP Hostscan." NFR gave us the attacker IP addresses we needed, but little else. For example, we could not see what port numbers were being attacked, which might have been useful in other contexts. NFR's GUI is also difficult in forensics mode: When you want to see the description for a particular event, you have to jump to another part of the GUI.

This test also exposed a problem common to all the products (except Barbedwire) - you can't see the offending packets. You never get to check the signatures to see if they are generating false positives.

Intrusion's forensics tool opens with a set of canned views into the forensics database: by attacker, by target, by priority and by signature group. We started with signature groups and clicked on the first level of the tree. Each major signature group was shown, along with a count of events. The group we were looking for stood out like a sore thumb, with hundreds of thousands of events. One more click (on "firewall services") and ICMP Ping Sweep and SMB Scan both stood out again - teaching us something we hadn't learned with NFR.

At this point, Intrusion doesn't further sort items, which means that if we went with the out-of-the-box product, we'd have to sort through long lists of events. But building a new tree was the quick solution to that. A few clicks let us add a new summarization level underneath signature and source IP address, and now we had the information we wanted. Sort of. We could see it on the screen, but there was no way to simply drop it into a spreadsheet.

ISS didn't let us quickly move to the signature we wanted, but gave us a few pre-built options. An obvious one, event analysis by attacker, created one line item for every "attacker" in the network. We sorted by count, and the infected machines on our network slipped to the top immediately. ISS would have let us trim the query by putting in only our corporate IP addresses, but because we had multiple sites, the ranges weren't compatible with its GUI.

From the list of attackers, we picked up our favorite feature in any of the products. We right-clicked on an attacker, and up came a list of questions you might like answered. In our case, it was "what events were generated by this attacker?" We clicked and wait 2 seconds, and then we knew what ISS was going to call the worm attackers.We right-clicked again on the relevant event, and there was our analysis question: "what are the sources of this event?" Another 10 seconds, and there was our list. Select the column, copy and there's the full list, exported. ISS got this right, in spades.

Cisco didn't have the slick response time of ISS, but did have similar features. In Cisco's event view, the basic paradigm is of a spreadsheet with movable and expandable columns. Grab whatever column you think is most important and drag it to the left, and Cisco's IDS Management Center will sort your data according to that column and summarize repeated items, giving a count along the way. It's a beautiful way to look at your data and would have been even better than ISS except for one flaw: When you move a column, you lose your place in the data.

Find something interesting and want to drag that column to the left to resort the data? Cisco does it, but whatever you were most interested in gets put back into the pile.

In our tests, ISS came though with flying colors, giving us the freedom to go through our data quickly, searching for patterns and problems. Cisco and, to a lesser extent, Intrusion both have a similar capability, but could learn a lot from the flexibility and freedom in the ISS interface.

Cisco and Intrusion are actually more flexible than ISS for some queries, because you can pivot on any column in its interfaces, whereas ISS limits you to the most common possibilities. However, in two months of working with these products, we never hit a wall with ISS.

Which IDS is right for you?

If you already know that you want an IDS, our two candidates would be ISS and NFR. If you think that writing signatures is going to be part of your application and if you're looking for a combination of policy enforcement and security, NFR comes in a clear winner with its N-Code language.

But if you think signatures should come from the vendor, ISS provided the best ability to manage the data thousands of systems would generate.

If you're not happy with ISS' options, Intrusion offers a similar product line but with reduced management capabilities.

Cisco has wide potential and enormous breadth in its sensor options. If the company is successful at integrating the CTR technology into its product, Cisco will be a clear contender. While its Web-based GUI is relatively slow, it also has some brilliant engineering behind it. Likewise, Intrusion has potential with its pieces but needs to build them into a better-integrated whole.

Barbedwire, the newcomer in this bunch, builds on the respected high-performance Snort engine. However, Barbedwire's choice of database and tuning on its hardware platform were major errors in execution. Performance problems on the low-speed networks we threw at its products suggest that it needs to go back to the engineering table. It could spend less time on elegant Linux GUI management pieces and more on the security application its platform is supposed to support.