Put Your Apps to the Test
- By Linda L. Briggs
- November 1, 2005
In many years of testing software application performance at a variety of companies,
quality engineer Carl Gerstle says he's seen nearly every application he's ever
tested fail-often at just 1 to 5 percent of its intended load. That means that
when he first tests an application intended for 1,000 users, "somewhere around
10 users, it falls over."
Gerstle, currently a principal quality engineer at a distributed Web and content
services company called Mirror Image Internet, worked on hardware, software
and design teams before becoming a QA engineer. "As smart as the people are
who design the code and the systems," Gerstle says, "the systems are complex
enough that if you don't get to see how they all interact before you field the
code, you're usually in for a horrible embarrassment."
That sort of commentary points to the need for application performance testing
software-along with the expertise to apply it correctly. Rapidly increasing
user loads, wildly complex software and distributed development that includes
worldwide outsourcing all ratchet up the need for more and better application
testing processes and tools.
Simulated software in the real world
Application performance testing software usually describes tools used during
and after product development to simulate user loads and ensure that an application
remains available and accurate under its anticipated use. The term also typically
includes performance optimization-analyzing code to see if there are patterns
causing it to perform poorly. While the speed and accuracy of Internet sites
is a big consideration today, in-house applications also need testing, especially
if they affect the business.
"Anything customer-facing should be tested," according to Gartner analyst Theresa
Lanowitz. That includes the Web sites used by help desks or order centers in
response to customer phone calls. When you call an airline or a catalog company
and hear, "Oh, the computers are slow today," Lanowitz says, "it's because they
haven't done proper performance testing. You might think [those applications]
aren't customer-facing, but in reality they are."
Purchasing a performance testing package, creating tests, running them and
interpreting the results is one way to beef up your software application testing
process. However, Lanowitz warns, a good software performance testing strategy
requires a degree of expertise that many companies lack. "The number of companies
with performance engineers on staff is really very small," she says. (See related
story, "Testing on the outside," below.)
With any kind of testing, of course, the rule is to begin testing as early
as possible, because study after study shows the cost of making a repair increases
exponentially the deeper developers are into the development lifecycle. "The
worse-case scenario," says Forrester analyst Carey Schwaber, "is testing [for
the first time] in production."
Testing on the outside
Even if companies regularly handle some testing functions internally, they may want to outsource larger tests. One advantage of outsourcing to a specialist is the savings in renting the extra bandwidth they’d need to adequately test a large application. According to Mike McCormick with AppLabs, for example, his company “already has infrastructure outside of the firewall, out in the cloud, at different data centers, that is used to drive the tests.” Mc- Cormick says AppLabs has some 550 megabits of bandwidth available across the U.S. for testing customer applications, and more can be added if needed.
There’s another advantage to outsourcing to a company that tests applications for a living: A good testing company has experts on staff who do performance testing for a living and test a range of applications for many companies. That can help companies with some of the considerable challenges of good performance testing, such as understanding and then applying the results.
Done correctly, performance testing isn’t simple, according to Gartner analyst Theresa Lanowitz. “It’s not something where anyone can take one of these tools and get a great bang for the buck.” Although writing the actual scripts typically isn’t difficult, she says, interpreting the resulting reports can be. “You have to figure out if [the results] are about the application or the infrastructure or the architecture. Performance testing is not something for the weak of heart.”
—Linda L. Briggs
Different jobs have different tools
Testing tools are generally priced in terms of the load simulated. Vendors sell
virtual user packs, in which cost is based on the size of the user load. Other
sorts of tests may overlap or fall within performance testing. Load testing,
for example, is generally a subset of performance testing; functional testing
compares actual performance to the functional spec. According to Schwaber, some
vendors offer some degree of integration between functional testing and performance
testing. That allows reuse of functional test scripts created earlier, as use
cases for performance testing. (See related story, "Right tool for the right
job," on below.)
At Mirror Image Internet, Gerstle's company, the QA group is responsible for
testing across a wide range of hardware and software platforms, including Windows
NT and 2000, Sun Solaris and various versions of Linux and Apache. Through huge facilities worldwide, the company provides
distributed Web and content services that include
streaming media and managed caching.
Gerstle uses a load and performance testing
tool—Segue Software’s SilkPerformer 6.5. The
product’s powerful dedicated scripting language
is a big draw for Gerstle, since it allows him to write test scripts natively, or produce them
through recording sessions, or use a combination.
Recording a sequence of keystrokes into a script for later playback is one
way to create test scripts, although it has limitations in flexibility. In Gerstle's
case, he often uses recorded sessions as a foundation "to get the actual flavor
of the protocol...what applications and servers are saying to each other and
to users." Later, he might shape the performance emulation script further by
hand-coding. When external code is needed, Gerstle says, his staff writes it,
then calls the code as a DLL from SilkPerformer.
Although SilkPerformer supports a wide array of protocols and can access all
layers of an enterprise application stack, that isn't its principal draw currently
for Gerstle. "In the space I'm working in now, I really only need to be able
to talk to databases and to talk over HTTP," he says. "To a large extent, I'm
testing either right at the database, [or] I'm communicating over OCI, or I'm
testing over HTTP and HTTPS."
Performance testing large loads
One solution for companies interested in introducing or increasing their application
testing is outsourcing the job. AppLabs Technologies, for example, is a global
IT services company specializing in software testing and development. Mike McCormick,
manager of partner solutions and business development, used RadView's WebLoad
to test applications for customers ranging from online retail businesses to
financial services to travel to news agencies.
AppLabs used RadView's product, for example, to test the MSNBC site for the
2002 Olympics in Salt Lake City, at loads exceeding 100,000 users. AppLabs uses
a variety of testing products from different vendors, selecting the appropriate
tool depending on the application under test. When the testing firm tried WebLoad
initially, it scaled well as virtual users were added.
language, rather than the sort of proprietary language that SilkPerformer uses.
on new people and get them up to speed," McCormick says.
more control of complex testing engagements, and the ability to read and write
to files. "That really enhances the capability of the tool," McCormick says.
"You can create a single test script that's flexible enough to represent multiple
users and react appropriately to dynamic data returned by the Web site." For
example, to simulate various users in applications that require logins, the
AppLabs staff can write test scripts that automatically insert unique user names
Right tool for the right job
Selecting the right performance and load testing tool for the job is something that AppLabs does on a regular basis. The global IT services company specializes in software testing and development, with customers such as American Airlines, InstallShield, Hewlett-Packard, Novell and SAP. The company tests applications for customers using a range of different tools from different vendors.
So what does AppLabs look for when selecting from the company’s range of available testing tools? Once the company knows the characteristics of the application they’ll be testing, according to Mike McCormick, manager of partner solutions and business development for the labs, the company first weighs whether a tool is applicable to a particular test. The product he selects to test a .NET application might be different from that used to load test a Citrix implementation.
Second, AppLabs looks at the functionality available in the tool, McCormick says. Does it provide the flexibility needed for the application under test? Third, he considers the stability of the product, including how many users the testing tool can scale up to and then sustain.
In making the final purchase decision on a testing product, the company also weighs ease of use, the availability of the tool—meaning the ease of working with the vendor on licensing—and finally, pricing.
—Linda L. Briggs
Somewhere there's a payoff
Iron Mountain saw the performance testing tool it installed 4 years ago pay
off in cost reductions and more. According to Michael Anthony, manager of corporate
systems' quality assurance for the records and information management company,
"we've seen a direct reduction in the amount of time and effort to test [systems.]"
Iron Mountain uses Mercury Interactive's LoadRunner for testing both Web-based
and character-based systems, including Oracle, Seibel and two proprietary systems.
Mercury's ability to handle the company's proprietary character-based systems
was an important determinant in selecting it, Anthony says. Another plus: The
product offers reports that can be customized with a high-level view for management,
or can allow network staff to drill down to the bits per second level if necessary.
Five years ago, Anthony says, Iron Mountain realized it needed a performance
testing system. The company grew rapidly over the past 10 years, to $1.8 billion
in gross revenue in 2004. Iron Mountain's large databases and diversity of systems,
Anthony says, made automating the testing process a must. "The databases that
we use are huge...There's a critical need for performance testing."
After ramp-up and training costs, LoadRunner has provided a solid return on
investment by reducing the testing labor pool, dramatically increasing how quickly
tests can be performed and increasing the sophistication of the testing process.
"Some load and performance testing just cannot be done in a practical way"
using manual testers, Anthony says. In addition, he's found that automated testing
is faster and more structured, producing results that can be documented and
repeated. If incremental changes are made to a system, for example, or a new
version is installed, the same performance test can be rerun, and results compared.
"It goes much more quickly than someone sitting at a terminal manually operating
the system," he says.
Using a performance testing tool also has forced more structured processes
on the company. Anthony and his staff run standard, repetitive tests regularly
against the Oracle and Seibel systems, for example, some as often as weekly.
"It's forced us to look at the updates we were doing and make sure our automation
scripts were in sync with our updates. It's a way to guarantee that incremental
updates to our production systems land in our regression test pool." (See related
story, "Lifecycle under new management," below.)
Lifecycle under new management
The growing focus on lifecycle management in software development is spilling over into performance testing. According to Forrester analyst Carey Schwaber, lifecycle management simply means determining what changes you make to an application and when you make them. With performance testing, it can mean managing the lifecycle so performance testing takes place at appropriate points in the lifecycle and is followed by appropriate changes back to the application.
For example, Schwaber says, a lifecycle management system that includes performance testing might mean QA engineers or performance testers use performance testing software to run a script, uncovering a performance problem. They send that information back through a defect tracking system to the developer. The developer then makes a change to the application. The change is tracked by the change management system and linked back to the original request from the performance testing software. Finally, the system notifies the performance test group that the problem was resolved and the software should be retested.
One vendor who does this well is IBM, according to Schwaber, with its Performance Optimization Toolkit. The toolkit includes Rational Optimization Developer as the IDE, ClearQuest as the defect tracking system, ClearCase as the version control system or software configuration management tool, and Performance Tester as the performance testing software. That suite comprises a lifecycle management system.
“The reason they’re able to do so well is because they also have very strong development tools,” Schwaber says. “IBM’s differentiator is they’re able to get down to the line of code instead of just the component.”
—Linda L. Briggs
Testing .NET Applications
One challenge with performance testing .NET applications is their complexity.
At Dataquick, a real property information firm, the QA staff uses an integrated
group of products from Empirix called e-TESTSuite for load testing and test
management of the company's large property information database. At the all-Microsoft
firm, applications are built in .NET and tested using an Empirix add-on module
specifically designed for .NET testing. The product "knew exactly what kind
of performance [issues] to look at in .NET apps," according to QA tester Glen
Pascual. "A lot of our developers were pretty impressed with how it tested .NET."
According to Pascual, the automation functions in the Empirix product make
it a good choice for his company. The suite includes e- Load, which Dataquick
uses for performance testing, load testing and stress testing. Using e- Load,
Pascual says, he can add more users or more load, while maintaining a dynamic
view of the testing process. "We could say, let's add 30 users and run it for
12 hours. If that passes, we can add more scripts to that test, and we can benchmark
Because Empirix uses the same scripting language for testing and monitoring,
scripts can be run over and over at different times. If a performance problem
crops up in production because of a software patch, for example, the original
test script can be rerun, and its results compared to earlier reports. The result
"helps development know where our problems are and where we need to focus,"
Before Empirix, Dataquick used a Windowsbased, stress-test application, Pascual
says. However, the staff found test results vague and hard to interpret. With
the Empirix products, he says, the graphical user interface tells him "exactly
what [I'm] looking at."
Testing gets some respect
The increasing complexity of software, driven in part by service-oriented architecture,
is starting to give performance testing the respect it deserves. At AppLabs,
which specializes in testing software for customers, McCormick says his company
has seen an increased buzz around performance testing during the last year or
"Performance has been one of those things that has been largely ignored" as
firms focused on features, functionality and getting product to market, Gartner's
Lanowitz says. During the height of the Internet craze, she says, "many dotcoms
died because performance was ignored." That's changing as companies realize
the business value of solid application performance. (See related story, "Don't
upset the developers," below.)
"There is a huge and ongoing need to do this kind of testing," Gerstle says.
Awell-behaved system, he says, shouldn't "go up in flames" when pushed to and
beyond capacity. Instead, "we [expect] performance to degrade gracefully as
load comes on. And hopefully, that degradation happens well beyond the expected
maximum that you want the system to perform at."
Don’t upset the developers
Assuming companies choose and implement the right application performance testing product for the job, their QA groups then face another big challenge— learning to work with the various developer groups whose products they’re essentially criticizing.
After all, testing a product’s performance and reliability often means pointing out the problems and sending it back for fixes.
At Mirror Image Internet, principal quality engineer Carl Gerstle fields a team of QA engineers in a one-to-one ratio with the development group—an unusually balanced ratio between the two groups for most companies.
Gerstle maintains harmonious relations with developers, partly by hiring technically savvy QA engineers, and by counseling junior developers on the art of collaboration, “so the developers understand that we’re not throwing rocks at them,” Gerstle says. “You have to get [beyond] the point of thinking ‘Part of my job is to upset developers,’ which it isn’t.”
It helps that Gerstle himself held a variety of technical jobs, including hardware development, software development, systems engineering and user interface testing. “I can look at a complex system from most angles, including the user’s. A good job of performance engineering includes empathy for the end user as well as [understanding] the system on which the code base is running,” he says.
—Linda L. Briggs