In-Depth

Put Your Apps to the Test

• By Linda L. Briggs
• November 1, 2005

In many years of testing software application performance at a variety of companies, quality engineer Carl Gerstle says he's seen nearly every application he's ever tested fail-often at just 1 to 5 percent of its intended load. That means that when he first tests an application intended for 1,000 users, "somewhere around 10 users, it falls over."

Gerstle, currently a principal quality engineer at a distributed Web and content services company called Mirror Image Internet, worked on hardware, software and design teams before becoming a QA engineer. "As smart as the people are who design the code and the systems," Gerstle says, "the systems are complex enough that if you don't get to see how they all interact before you field the code, you're usually in for a horrible embarrassment."

That sort of commentary points to the need for application performance testing software-along with the expertise to apply it correctly. Rapidly increasing user loads, wildly complex software and distributed development that includes worldwide outsourcing all ratchet up the need for more and better application testing processes and tools.

Simulated software in the real world
Application performance testing software usually describes tools used during and after product development to simulate user loads and ensure that an application remains available and accurate under its anticipated use. The term also typically includes performance optimization-analyzing code to see if there are patterns causing it to perform poorly. While the speed and accuracy of Internet sites is a big consideration today, in-house applications also need testing, especially if they affect the business.

"Anything customer-facing should be tested," according to Gartner analyst Theresa Lanowitz. That includes the Web sites used by help desks or order centers in response to customer phone calls. When you call an airline or a catalog company and hear, "Oh, the computers are slow today," Lanowitz says, "it's because they haven't done proper performance testing. You might think [those applications] aren't customer-facing, but in reality they are."

Purchasing a performance testing package, creating tests, running them and interpreting the results is one way to beef up your software application testing process. However, Lanowitz warns, a good software performance testing strategy requires a degree of expertise that many companies lack. "The number of companies with performance engineers on staff is really very small," she says. (See related story, "Testing on the outside," below.)

With any kind of testing, of course, the rule is to begin testing as early as possible, because study after study shows the cost of making a repair increases exponentially the deeper developers are into the development lifecycle. "The worse-case scenario," says Forrester analyst Carey Schwaber, "is testing [for the first time] in production."

Different jobs have different tools
Testing tools are generally priced in terms of the load simulated. Vendors sell virtual user packs, in which cost is based on the size of the user load. Other sorts of tests may overlap or fall within performance testing. Load testing, for example, is generally a subset of performance testing; functional testing compares actual performance to the functional spec. According to Schwaber, some vendors offer some degree of integration between functional testing and performance testing. That allows reuse of functional test scripts created earlier, as use cases for performance testing. (See related story, "Right tool for the right job," on below.)

At Mirror Image Internet, Gerstle's company, the QA group is responsible for testing across a wide range of hardware and software platforms, including Windows NT and 2000, Sun Solaris and various versions of Linux and Apache. Through huge facilities worldwide, the company provides distributed Web and content services that include streaming media and managed caching.

Recording a sequence of keystrokes into a script for later playback is one way to create test scripts, although it has limitations in flexibility. In Gerstle's case, he often uses recorded sessions as a foundation "to get the actual flavor of the protocol...what applications and servers are saying to each other and to users." Later, he might shape the performance emulation script further by hand-coding. When external code is needed, Gerstle says, his staff writes it, then calls the code as a DLL from SilkPerformer.

Although SilkPerformer supports a wide array of protocols and can access all layers of an enterprise application stack, that isn't its principal draw currently for Gerstle. "In the space I'm working in now, I really only need to be able to talk to databases and to talk over HTTP," he says. "To a large extent, I'm testing either right at the database, [or] I'm communicating over OCI, or I'm testing over HTTP and HTTPS."

Performance testing large loads
One solution for companies interested in introducing or increasing their application testing is outsourcing the job. AppLabs Technologies, for example, is a global IT services company specializing in software testing and development. Mike McCormick, manager of partner solutions and business development, used RadView's WebLoad to test applications for customers ranging from online retail businesses to financial services to travel to news agencies.

AppLabs used RadView's product, for example, to test the MSNBC site for the 2002 Olympics in Salt Lake City, at loads exceeding 100,000 users. AppLabs uses a variety of testing products from different vendors, selecting the appropriate tool depending on the application under test. When the testing firm tried WebLoad initially, it scaled well as virtual users were added.

For AppLabs, another plus with WebLoad is its use of JavaScript as a scripting language, rather than the sort of proprietary language that SilkPerformer uses. Use of a well-known language such as JavaScript "makes it easier for us to bring on new people and get them up to speed," McCormick says.

Since JavaScript supports regular expressions, it gives the AppLabs engineers more control of complex testing engagements, and the ability to read and write to files. "That really enhances the capability of the tool," McCormick says. "You can create a single test script that's flexible enough to represent multiple users and react appropriately to dynamic data returned by the Web site." For example, to simulate various users in applications that require logins, the AppLabs staff can write test scripts that automatically insert unique user names and passwords.

Somewhere there's a payoff
Iron Mountain saw the performance testing tool it installed 4 years ago pay off in cost reductions and more. According to Michael Anthony, manager of corporate systems' quality assurance for the records and information management company, "we've seen a direct reduction in the amount of time and effort to test [systems.]"

Iron Mountain uses Mercury Interactive's LoadRunner for testing both Web-based and character-based systems, including Oracle, Seibel and two proprietary systems.

Mercury's ability to handle the company's proprietary character-based systems was an important determinant in selecting it, Anthony says. Another plus: The product offers reports that can be customized with a high-level view for management, or can allow network staff to drill down to the bits per second level if necessary.

Five years ago, Anthony says, Iron Mountain realized it needed a performance testing system. The company grew rapidly over the past 10 years, to $1.8 billion in gross revenue in 2004. Iron Mountain's large databases and diversity of systems, Anthony says, made automating the testing process a must. "The databases that we use are huge...There's a critical need for performance testing."

After ramp-up and training costs, LoadRunner has provided a solid return on investment by reducing the testing labor pool, dramatically increasing how quickly tests can be performed and increasing the sophistication of the testing process.

"Some load and performance testing just cannot be done in a practical way" using manual testers, Anthony says. In addition, he's found that automated testing is faster and more structured, producing results that can be documented and repeated. If incremental changes are made to a system, for example, or a new version is installed, the same performance test can be rerun, and results compared. "It goes much more quickly than someone sitting at a terminal manually operating the system," he says.

Using a performance testing tool also has forced more structured processes on the company. Anthony and his staff run standard, repetitive tests regularly against the Oracle and Seibel systems, for example, some as often as weekly. "It's forced us to look at the updates we were doing and make sure our automation scripts were in sync with our updates. It's a way to guarantee that incremental updates to our production systems land in our regression test pool." (See related story, "Lifecycle under new management," below.)

Testing .NET Applications
One challenge with performance testing .NET applications is their complexity. At Dataquick, a real property information firm, the QA staff uses an integrated group of products from Empirix called e-TESTSuite for load testing and test management of the company's large property information database. At the all-Microsoft firm, applications are built in .NET and tested using an Empirix add-on module specifically designed for .NET testing. The product "knew exactly what kind of performance [issues] to look at in .NET apps," according to QA tester Glen Pascual. "A lot of our developers were pretty impressed with how it tested .NET."

According to Pascual, the automation functions in the Empirix product make it a good choice for his company. The suite includes e- Load, which Dataquick uses for performance testing, load testing and stress testing. Using e- Load, Pascual says, he can add more users or more load, while maintaining a dynamic view of the testing process. "We could say, let's add 30 users and run it for 12 hours. If that passes, we can add more scripts to that test, and we can benchmark it."

Because Empirix uses the same scripting language for testing and monitoring, scripts can be run over and over at different times. If a performance problem crops up in production because of a software patch, for example, the original test script can be rerun, and its results compared to earlier reports. The result "helps development know where our problems are and where we need to focus," Pascual says.

Before Empirix, Dataquick used a Windowsbased, stress-test application, Pascual says. However, the staff found test results vague and hard to interpret. With the Empirix products, he says, the graphical user interface tells him "exactly what [I'm] looking at."

Testing gets some respect
The increasing complexity of software, driven in part by service-oriented architecture, is starting to give performance testing the respect it deserves. At AppLabs, which specializes in testing software for customers, McCormick says his company has seen an increased buzz around performance testing during the last year or so.

"Performance has been one of those things that has been largely ignored" as firms focused on features, functionality and getting product to market, Gartner's Lanowitz says. During the height of the Internet craze, she says, "many dotcoms died because performance was ignored." That's changing as companies realize the business value of solid application performance. (See related story, "Don't upset the developers," below.)

"There is a huge and ongoing need to do this kind of testing," Gerstle says. Awell-behaved system, he says, shouldn't "go up in flames" when pushed to and beyond capacity. Instead, "we [expect] performance to degrade gracefully as load comes on. And hopefully, that degradation happens well beyond the expected maximum that you want the system to perform at."