News

Fortify Tracer Shines a Light in the Black Box

• By John K. Waters
• October 30, 2006

Black-box testing is standard practice for analyzing the security of deployed Web applications. It's a practice that falls short in key areas, however, making it difficult for developers to find and repair code flaws.

A report, "Taking the Blinders off Black Box Security Testing," cites several drawbacks to black-box testing as currently practiced. Among other findings, the researchers concluded that manual and automated black-box security tests of Web applications generally reach less than 50% of security-critical sites within the code. The tests also failed to pinpoint the location of vulnerabilities, and to test all input sources. The research was sponsored by Fortify Software.

The black box security testing report is based on sixty days of data gathered using a pre-release version of the company's new Tracer tool. Fortify Software used the tool to monitor black-box security tests on numerous applications varying in function, size, and complexity, says Barmak Meftah, vice president of products and services.

"We found that black-box testing predominates in the area of quality assurance and audit," Meftah explains. "So we decided to look at this [type of testing] and see whether it is complete; whether, when these tools run, they are giving an accurate gauge of the security of an application. We found that it's a good exercise, but it's an incomplete exercise."

Black-box testing, which is sometimes called "ethical hacking," is a crucial part of securing applications, because it's used to identify vulnerabilities that can be found only when an app is running. Testers probe a computer system or network to seek out vulnerabilities that an attacker could exploit. This type of testing operates from outside the application, so it provides no understanding of the application’s source code — the app is a black box.

"All organizations do some kind of black-box testing or ethical-hacking exercise," says Meftah, "whether as part of their audit exercise or part of their quality assurance run. But it's very difficult for the developers charged with fixing the vulnerabilities. They get a report that says they failed a black-box exercise, but they get little code-level information about the issue in terms of why it happened and where it happened in the code. The audit team tells them how many issues they have, but not how much of the application was hit to find those issues."

Palo Alto, CA-based Fortify, which focuses on software security at the application layer, has just made its Fortify Tracer tool generally available. Tracer is designed to work in conjunction with a black-box test or ethical hacking exercise to report on coverage percentages and code-level details about runtime security errors discovered during the tests.

At the heart of the product is Fortify's Call Site Monitor technology, which tracks security-critical APIs, such as database and file system, within the Web application itself, and detects runtime vulnerabilities that are not visible through an application’s Web interface.

Other features in the new product include:
• Security coverage reports that detail the percentage of security-critical functions exercised during a test. Key areas of the application that interact with sensitive interfaces, such as Web input, the database, and the file system, are tracked separately to provide additional coverage information.
• Dashboards that communicate key metrics and allow users to compare runs, inspect issues, and find the flaws.
• The product works on any Java EE executable (.war/.ear) files; users simply point to the file and the Fortify instrumentation engine inserts monitors at security-critical call sites.
• Detailed reports that show vulnerabilities according to their categories, such as cross-site scripting and SQL injection.

Security expert and author Gary McGraw has famously referred to black-box testers as "badness-ometers" that yield very limited information. "When you do a black-box test, you're sort of firing this bullet into the software through the front door," McGraw says. "All you really know is that it went down in there somewhere and something bad happened. You're like Luke Skywalker shooting that thing into the Death Star. It blows up, but you don't really know why."

McGraw is CTO of Cigital, a Northern Virginia-based software security consulting firm, and he serves on Fortify's advisory board.

"Tracer is about helping you to diagnose the problem, instead of just letting you know that you're in trouble," McGraw adds. "It's a great segue way from the current practice of relying on your badness-ometer to actually doing something about software security. It shows you which part of the application is blowing up so that you, as a developer, can build a better Death Star."