Load testing: A developer's crystal ball
- By Sandra Taylor
- July 31, 2001
The reason to load test your client/server application is to detect multi-user interaction errors, right? Wrong! If this is your view of load testing, you are seeing only one dimension of this multifaceted tool. Consider the following.
More than a few client/server systems have been blind-sided by scalability issues. The term "hitting the scalability wall" evolved from the far too common occurrence of a client/server application's inability to accommodate a growing number of users. While proponents of a three-tier model will argue its alternative merits, the truth of the matter is that it is virtually impossible to predict if and/or when any client/server application (two, three or n-tier) will begin to degrade and whether a given configuration can support the desired number of users.
Most discussions on scalability deal with technological reasons for a degradation. What tends to be ignored is the impact the degradation has on the business. As more and more client/server applications can rightfully be termed mission-critical, the impact of systems that fail to scale becomes more acute. Research conducted by Software Productivity Group, Natick, Mass., in 1995 showed that nearly 31.1% of the Fortune 1000 respondents to our survey were using client/server systems for critical business applications. In 1996, that number had risen to 41.1%. Client/server is growing up -- the applications are more central to the heart of the business -- and the stakes are getting higher.
Some companies have already discovered the one way to determine if their system will scale or fail. These organizations are using automated load testing tool suites that provide a virtual software crystal ball into an application's performance characteristics before the application is deployed. Not only can these organizations graphically see the characteristics of an application as it is being implemented, they also can look into the future and see what an increasing load will do to the applications -- and more importantly, to the business functions and users of that application.
Load testing in the financial services sector
Physically moving a multimillion-dollar-a-day financial trading
application in the space of a weekend is not for the faint-hearted.
Add the unknowns of a totally new environment -- new terminals,
new LAN, new servers, new database -- and the move takes on another
dimension of complexity and pressure. Such a move was undertaken
recently by a large Wall Street securities-brokerage firm, which
ADT was asked not to identify.
In the firm's preparation for the combined physical move/database
transition, each major system component was attended to by its team
of experts -- database experts, networking experts, operating system
experts and other experts. Each subsystem had to be checked, re-checked,
and checked again. Yet in this system, as in the vast majority of
client/server systems, there was no unifying element that could
pull everything together and orchestrate a full-bore system level
Real-Time Technology Solutions (RTTS), a New York City-based consulting
firm specializing in automated testing software, was initially brought
in to develop a test methodology, then implement and deploy SQA's
Team Test automated testing system. According to Bill Hayduk, President
of RTTS, he started the project looking for application errors.
As the project advanced, he and the other members of the development
team found that load testing offered far more than multi-user error
detection. It provided the basis for consistent, repeatable system
level testing. It is one thing to run a subsystem test to validate
that records can be read from and written to the database. It is
another thing to validate transactions and responses when a few
users are driving the application from their PCs. It is yet another
plateau to systematically emulate an increasing number of users
performing a variety of typical operations and graphically watch
response times and system resource utilization. Said Bill Hayduk,
"We started the project looking for errors. We ended with a framework
for an integrated system level test."
Following the close of trading on a Friday afternoon, the system
was powered down and moved to its new home. The following Monday
morning, a switch was toggled and the system went live. To date,
no major errors have been detected.
Orchestrating an environmental test
The subsystems of a client/server system are like an orchestra. If each component
is tuned and working well with the other subsystems, the result is harmony.
If only one subsystems is off, the result is discord.
Load testing introduces a conductor into the picture -- a conductor that can
direct the operation of the entire environment. If an aberration develops and
a certain part of the score needs work, test cases that isolate the errant operation
can be run again and again until the problem is solved and the system is functioning
properly and within specs.
Just as a conductor cannot guarantee that a member of the orchestra will not
make some unexpected faux pas, automated testing software cannot guarantee error-free
code. However, it can trap -- and has in an increasing number of high-profile
installations -- many of the show stoppers that lie in wait in today's complex
client/server systems. Equally important, it can also provide the knowledge
and confidence that the system will function under a growing user load.
Today's load testing tool suites generally take one of two approaches. Each
approach involves the use of test scripts. Scripts can be generated by recording
actual user input, programmed using a test tool's scripting language, or generated
using a combination -- first recording actual user input and then modifying
those scripts with an editor provided with the test tool. Executing a script
is essentially (and simply) the same as a user sitting at a PC and interacting
with the application.
Single test per processor
In one approach to load testing, the load is
driven by test scripts, which are generated by recording actual
user input, executing on multiple PCs.
Source: Software Productivity Group
Multiple test per processor
Test scripts programmatically drive the application
from the PCs essentially in the same manner as multiple users would
drive an application in a live production environment.
Source: Software Productivity Group
The above graphic is an example of PCs executing
individual scripts and the multithreaded systems executing multiple
Source: Software Productivity Group
In one approach to load testing, the load is driven by these test scripts executing
on multiple PCs. (See Fig. 1
.) In the other approach, the load testing
software drives multiple instances of the tests on a single system. (See Fig.
The multiple PC approach offers the most realistic approach to system load
testing. Test scripts programmatically drive the application from the PCs in
essentially the same manner as multiple users would drive an application in
a live production environment. While this approach is intuitively sound, the
reality of the matter is that it is impracticable to build a test system that
reflects a full-blown production configuration. Modern day client/server systems
comprise hundreds, and in some cases thousands, of PC clients. Physically amassing
even a small percentage of those PCs would be a monumental task. For testing
such high-end systems, the simulation approach to load testing is more appropriate.
If the application demands more simulated users than can be accommodated on
one system, multiple systems -- each running multiple virtual users -- can be
added. It is also not unusual to see a combination of the approaches with some
PCs executing individual scripts and the multithreaded systems executing multiple
simulated users. (See Fig. 3.)
Initially, test tool vendors offered just one of the testing capabilities.
The single-test-per-PC camp held that performance characteristics could be determined
and then interpolated to ascertain the results of adding more users. The simulated
user camp retorted that interpolation is not reliable when talking about high-end
systems of tremendous complexity. The reality is there is a valid case for both
approaches. In fact, it is now common for vendors to offer both capabilities.
Some companies provide internally built product suites, while others augment
product sets through alliances. The two leaders in the field today are Mercury
Interactive Corp., Sunnyvale, Calif., and the McLean, Va.-based Performix unit
of Pure Atria, Sunnyvale, Calif. Mercury Interactive offers virtual user testing
via its LoadRunner product suite. Performix, acquired by Pure Atria in 1995,
focuses on simulated load testing. Pure Atria will add a GUI-based system from
technology recently acquired with Integrity QA. Those plans have yet to be disclosed.
Newton Centre, Mass.-based Segue Software Inc. and Burlington, Mass.-based SQA Inc. had
initially offered their own PC-based load-testing facilities (in addition to a merger agreement with Rational Software Corp., Santa Clara, Calif.). By the time this article appears in print, Segue Software Inc. will have followed suit with its own simulation facility.
Other vendors offering various load testing facilities include companies such as: the Dallas-based AutoTester unit of Software Recording Corp.; Compuware Corp., Farmington Hills, Mich.; Performance Awareness Corp., Raleigh, N.C.; and Softbridge Inc., Cambridge, Mass.
More than one nugget
Load testing tool suites can offer a veritable gold mine of benefits. The list below identifies only a handful of possibilities. Note that the terms used below may vary from vendor to vendor. The important issue is the range of capabilities these tools offer and the benefits they can bring to the test tool user.
Indeed, determining the focus of a test is one of the challenges facing organizations that opt to use the tools. Every vendor offers classes on how to use their respective toolsets. But the methodology of automated testing -- what to test versus the mechanics of using the tool suite -- is still a relatively new discipline. These tool suites offer so much power and scope that the most appropriate approach is not always clear to an automated testing neophyte.
Multi-user testing This is the classic error detection functionality. Today's client/server systems have an extraordinary number of moving parts. Not only must each part work correctly on its own, they must work together in an extremely complex environment involving application, operating system, middleware, networking and database software. Unfortunately, what may work correctly with only a few users can fall apart under the load of several users. By stressing the system with multi-user testing, many of these insidious errors can be uncovered.
Architectural validation Many organizations are currently in the process of developing computing architectures that will carry them into the next century. It is one thing to talk of strategies and products. It is another thing entirely to put the architecture into action -- to select the products, and then build the first application that can make or break the architecture. One company is well on its way in this process -- Gap Inc., a $4 billion clothing retailer.
Load testing and Gap Inc.
With a chain of over 1700 customer-oriented retail stores, Gap
Inc. has made its mark in the retail industry. Like many organizations
of this size, the company is looking at its computing architecture
and beginning the process of developing an architecture that will
span its computing environment. According to Phil Wilkerson, director
of Technical Architecture, Gap Inc. first had to develop its strategy
and then select the components to implement that strategy. Core
elements of the company's strategy include a three-tier client/server
model, a move to object technology, plus an ability to access corporate
data in an MVS-based mainframe environment. For the mainstay system
components, Gap selected Sun from Sun Microsystems Inc., Mountain
View, Calif., for its server engines, Forté from Forté
Software Inc., Oakland, Calif., for the application development
tool, Tuxedo for the transaction processing monitor and Informix
for the database.
With many of the development components selected, Wilkerson believed
there was one more piece to the puzzle -- automated testing software
that would objectively manage the system level tests and would prove
the architecture could handle the load. According to Wilkerson,
"We needed a mechanism to reduce the fear factor -- to control the
Mercury Interactive's automated testing tool suite was selected
as that mechanism and put to work on a series of tests. The testing
was a success, according to Wilkerson. The Mercury tool suite allowed
Gap to test whether "the client/server monster" is functioning properly
end-to-end, from the interfaces to network connectivity to the build
process to the business process. Validating the architecture was
not the only hard payback Gap received from these tools, he said.
The retailer said the tools also discovered a memory allocation
and a RAID channel problem. As a result of system readjustments,
the end-to-end response times became phenomenal -- sub-second response
time on a test query into a 40,000-product database. Having this
kind of information enables an organization to scale networks and
hardware to handle production loads accessing huge amounts of information.
Configuration tuning Not only are there more discrete pieces in today's
systems, each component alone often involves complex configuration options that
can ripple through a system and ultimately affect response time. Manual testing
cannot provide the consistent and repetitive testing environment needed to truly
stress a system and objectively capture the performance metrics. Load testing,
on the other hand, can take configuration optimization from a subjective realm
into a world of objective charts and graphs that show the effect of various
configuration optimizations. Another issue here is whether a performance bottleneck
is the result of a problem with the client/server model (such as two-tier versus
three-tier) or because of a hardware configuration problem. At Gap, for example,
the hardware caused the problem and a modification to the configuration had
a very positive impact on system performance.
System upgrade comparisons Some of the most interesting aspects of
the growing acceptance and maturity of the load testing market are the various
ways companies use these tools. One of the most resourceful uses involves a
system upgrade "shoot-out."
With the flexibility in and competitiveness between today's hardware systems,
choosing from the available options for a system upgrade can be a daunting task.
Load testing can provide an objective measure of these options -- essentially
using the very application that will run on the upgraded hardware. Not only
can a company see what the upgrade will (or will not) do in terms of response
time for the existing number of users, the company can increase the number of
users to determine how much headroom the new hardware offers before another
upgrade is required.
User-acceptance testing Always a relatively sticky issue in terms of
acceptance, load testing software provides a mechanism for demonstrating to
the acceptance team how a system performs under load. Load testing can also
provide feedback to a development team and can be used to measure how the team
is doing against any performance requirements defined by the system specifications.
System software revision verification Some of the more interesting times in a development group can occur when it upgrades an application to a new version of a vendor's packaged software. Whether an operating system, middleware or other system software component, a new software version can cause havoc in an application that had been working perfectly. Load testing software can provide a safe pre-deployment venue for discovering multi-user errors and can compare performance characteristics with those from the previous version. With the time, effort and expense involved in deploying client/server systems, the last thing an organization wants to do is distribute an application that is inferior to an earlier version.
Not a free lunch
Staffing There is tremendous potential within load testing tool suites. There is, however, a price to pay financially, and perhaps more importantly in terms of organizational operations and infrastructure. The purchase price of these tool suites is the tip of the iceberg.
The staffing issue is often one of the hidden land mines in automated testing, and the majority of companies underestimate the quality of staff needed to develop and manage an automated testing effort.
Companies have come to understand that implementing a client/server system is a non-trivial exercise. Testing such systems is no less complex. Therein lies the surprise for many organizations. Test tool suites are software systems in their own right and require all the resources necessary to care for and feed such a system. The skills present in a manual quality insurance (QA) group do not meet the needs of an automation-based group and must be augmented with software development skillsets.
Many companies that have made the leap to automated testing now recognize this fact of life and have turned to a growing number of third-party testing experts for help with their first project. While this strategy requires an additional up-front expenditure, it can be the most effective way to:
- acquire intelligence about the methodology of automated testing;
- eEstablish a formal automated testing program; and
- test an application.
Infrastructure This aspect may be the most difficult area for many companies to tackle. In many organizations, automated testing tools fall under auspices of a QA group. In spite of the emphasis on quality, the QA function is frequently relegated to the bottom of the development food chain. For automated testing tools to be effective, such thinking must change. Testing tools work best when integrated into the development cycle from the start. As development progresses, testing can often identify issues early on, before changes become a major undertaking. To do this, the QA and development groups must work hand-in-hand. With a premium on development groups, this can be a fairly significant barrier.
The Web -- Let the game begin!
Most end-user, client/server systems are currently intra-company; that is, the users are typically employees of that company. As such, any dissatisfaction regarding performance is within the confines of the company, and solutions can be negotiated within the realm of internal politics and corporate memos.
As companies expose their business systems to the public, performance will become a critical factor. And we need not be talking about high-profile business critical applications. Someone just browsing for information will soon become frustrated with a lethargic system. That company, in the brief space of a mouse click, can lose a customer. With serious buy-sell business applications and the Web's ability to even the competitive playing field, the issue is magnified. With such applications, the question is not if you should consider these tools, but when and which ones.