- By Philip E. Courtney
For IT managers, testing was never easy. But in the age of the
Internet, it just got harder. Once you had your hands full testing for all the possible things that could happen
within your company. Now your "users" come from all parts of the world, and they want their systems to
work as easily and reliably as the family television. Suddenly, for IT, the world is like a living room. And things
had better work.
But making things work in such a complex setting is daunting. "There are so many intertwined and interconnected
items within a Web-based or e-commerce system that it's impossible to say any one discipline is less important
than another," said David Richardson, chief technology officer at Rhino Productions Inc., a Tempe, Ariz.-based
firm that develops custom e-commerce and Web applications. "Testing and QA disciplines must focus on the individual
components, the entire infrastructure and how the system is used by the customer."
This is because most Web sites contain an underlying collection of diverse technologies that have been integrated
to meet a variety of business requirements. For example, Acton, Mass.-based MotherNature.com is an e-commerce company
that sells more than 30,000 natural health-care products in 1,400 different categories. The company uses a combination
of tools found in the e-Test product suite from Watertown, Mass.-based RSW Software Inc. to ensure the accuracy
of systems distributed across a multitude of daisy-chained servers and connected by a variety of networking technologies.
MotherNature.com has assembled quality assurance and testing processes for individual components within the application,
as well as load testing
for approximately 1,250 simultaneous, simulated customers.
Following the evolution of the Web from simple, online marketing brochures to full-fledged centers of e-commerce
-- the Internet commerce application market will soar to $1.7 billion this year according to International Data
Corp., Framingham, Mass. -- are dozens of testing tools designed to help ensure the quality of Internet applications.
Although these products support a variety of Internet testing disciples, companies that have made heavy investments
in Web-based applications state that the most important tools in their arsenal are those designed to assist with
application, component and load testing.
In a variety of very public failures, a number of high-profile e-commerce sites have shown what can go wrong
when systems are changed and testing takes a back seat. These development issues are, naturally, just as pressing
at sites like MotherNature.com.
"Each product in our system has its own Web page connected to a data bank containing the unique ID and
UPC code," said Vic Salemme, director of quality assurance at MotherNature.com. "We need to test customer
access to each page and validate the ability to link products, codes and prices on an individual basis. We also
need to ensure quality performance and eliminate bottlenecks during expected increases in volume."
The Web was once the exclusive domain of marketing. In recent years, the thrust toward e-commerce has served
to meld marketing and development requirements. This has helped instill traditional system development life-cycle
methodologies for the creation of Web applications.
For testing, however, the question arises as to when disciplines and tools should begin to be used within the
process. Without variation, the answer is once the requirements have been defined.
For instance, new Web development activities typically require two primary tasks. The first, driven by marketing
types, is to build the look and feel -- or storyboard -- requirements. The other task, which is more development-oriented,
is to build the internal design requirements. The completion of these tasks can produce a prototype that models
the look and feel of the site from a user perspective, as well as the flow of interaction using the underlying
"You've got to try to get the process down first before applying tools," explained Pete LeFebvre,
vice president of development at Gallagher Financial Systems, Coral Cables, Fla., a provider of mortgage origination
and pipeline management software. "Applying tools too early during development activities can be counterproductive.
Instead, the tools should be applied where they fit within the process."
Added Rhino's Richardson, "Test scripts and scenarios can be created once issues such as presentation,
navigation, data design, data transfer and so on, have been resolved." And creating the scripts and scenarios
is where the tools come into play.
Creation of test scripts for Web applications typically occurs as development is nearing completion and takes
the form of regression testing. Nearly identical to the disciplines used for validating the GUI screens of client/server
applications, regression testing for a Web-based system is one of the most fruitful methods for pinpointing errors
as early as possible prior to a production implementation.
The caveat, however, is that the capture/playback tools used for regression testing must be capable of recognizing
the differences between a traditional client/server application and a Web application.
One major difference is that, while standardized screens are common in the client/server environment, a user
can access a Web application through any number of different browsers running on any number of different operating
systems. Today, for example, both Microsoft and Netscape each have at least three predominant versions of their
browsers on the market -- and what works in one may not necessarily work in another.
"Internet Explorer 3.0 doesn't work very well with Active Server Pages [ASPs] or Java, but we have few
problems with IE 4.0 or IE 5.0," said Kathy Kirk, QA lead at custom software and e-commerce development firm
Key Tech Inc., Eagan, Minn. "We need to be able to test for a multitude of browsers and gain an understanding
of what works and what doesn't work." To support its different application and load testing requirements,
Key Tech uses Web- Tester from Cyrano, Newburyport, Mass.
In addition to browser differences, the look and feel of today's Web screens are always subject to change even
though functionality remains the same. Using Active Server Pages, Java applets, ActiveX and other components to
produce date/time stamps, banner pages, animation, tickers and other dynamic effects, a site accessed by a user
in the morning can appear dramatically different by afternoon -- without a change in functionality. Accurate testing
dictates that the capture/playback tools used for Web testing recognize and accommodate changes in appearance while
also recognizing and pinpointing changes in functionality.
"You can't build a Web regression test based on what's on a screen," said Michael Maggio, senior vice
president and general manager at Segue Software, Lexington, Mass. "Compared to GUI testing, the screens and
the controls are very different."
Just like in GUI testing, the capture function in a regression-testing tool is used to automatically generate
the scripts that will be used for playback. And while these scripts will be used for application testing, it is
equally important that they remain reusable and flexible so that they can be employed again later in the cycle
during the various scenarios established for load and performance testing.
"The scripts created early in the process must be reusable when the time comes to perform simulated load
testing," said Salemme at MotherNature.com. "[This] avoids rework and allows different scripts to be
used at different times by different virtual users when validating system performance."
Lastly, though, it is important that scripts provide support for one of the most prevalent challenges in testing:
test data generation. As Salemme points out, "Our largest challenge was data banking all of our products so
that they could be read by our scripts."
Support for test data comes with the ability to separate captured data, such as user entry and tables, from
the scripts and place it into a file. During playback, the script will read the data from the created file -- or
any other file. This functionality allows testers and QA analysts to alter test cases without altering a script.
"The best data comes from the real application," said Yves de Montcheuil, director of Web solutions
at Cyrano. "For new test data generation some of the tools most frequently used are TestBytes [from Platinum
Technologies Inc.] and DataShark [from HardBall Software Inc.]."
But what happens when an application component fails during a regression test? In addition to basic HTML, today's
Web-based applications comprise numerous controls, Java applets, Perl scripts and more. Underlying languages can
be anything from C/C++ to Visual Basic.
While many problems in e-commerce and Web-based applications can be traced to performance issues, other failures
are sometimes traced to problems within the underlying code -- code that must be analyzed and debugged. And in
many cases, that code was developed on one virtual machine for use on many others that use different run-time characteristics.
Other problems can be traced to memory leaks, stack errors or incorrect usage of APIs and links -- or just plain
old syntax errors in the employed programming language.
But one of the biggest challenges at the component level is the variety of languages, platforms and environments
used for development. While Visual Basic development typically occurs on the client-side of the house, Java developers
typically focus on the server, and C++ can be found anywhere and everywhere; this joining of so many different
individual components can wreak havoc when the sum of the parts become whole as a Web application.
"Without care in development and testing, the instantaneous worldwide deployment of an application can
also mean the instantaneous worldwide deployment of bugs," said Doug Carrier, product manager for Java development
tools at Compuware Corp.'s NuMega Labs, Nashua, N.H. "Problems typically occur when moving to a three-tier
environment from a two-tier and the multiple components increase testing complexity."
While there a number of tools available to assist in pinpointing these problems, Rhino Productions' Richardson
points out that developers must exercise a great deal of care when performing modifications at such a low level
within the application. "The components and the programming languages are the lowest common denominator in
most of these cases," he said. "Although it's important to make sure the problem is corrected, it's also
important to test not only the single component, but the remainder of the application. The integrated infrastructure
of the Web means that the one changed component can be used in any number of other locations within the application."
Others, however, have seen little need for the tools used to assist in debugging. They profess that most errors
occur during times of high traffic, and that a proper design helps avoid errors at the lowest level.
"There are so many different pieces in a system," said Kirk at Key Tech Inc., "that there's really
no justification in performing a line-by-line analysis of the code. We have enough trapped code available to perform
a review of a problem if needed, but we've found that most problems occur in the infrastructure of the system."
Validating the infrastructure
Of all the disciplines invoked for validating Web-based applications, load testing has to be listed as the most
crucial. As an interactive medium in a society grown accustomed to instant gratification, users and customers expect
an almost immediate response when doing business over the Web. Patience is not a virtue, and a slow site results
in lost business.
"Customers want and expect a reasonable response time," said Kirk. "They're most interested in
But load testing is almost like 'black magic' in terms of Web testing because it requires companies to identify
both real and theoretical limits to create the proper scenarios. And the Web is a fast moving target. Still, this
discipline helps companies identify, isolate and analyze the items that can have a detrimental effect on response
time and performance, including configurations for networks and servers, communications and bandwidth, or just
plain old volume.
When performed early enough in the development process, load testing allows organizations to remedy internal
configuration issues prior to a production roll-out.
"We identified a number of configuration and resource issues during one early load test when we hit 30
concurrent users and the system just quit," explained LeFebvre at Gallagher Financial Systems. For load testing,
his company employs LoadRunner from Mercury Interactive, Sunnyvale, Calif. Last month, Mercury released Version
6.0 of that product.
"Load testing is a matter of creating different scenarios to ensure proper performance," said Tony
Troppito, director of software quality assurance at Burlington, Mass.-based Open Market, a company that produces
software for commerce service providers. Open Market uses products from Segue's LiveQuality suite. "You've
got to test for different configurations and different volumes of users. You've got to review specific situations
such as 'How will this system handle 10,000 transactions per hour while another system is handling 5,000?'"
An important function needed in a load-testing tool is the ability to create scenarios as well as virtual users.
While scenario creation allows QA analysts to establish different testing situations for a multitude of hardware,
network and communications configurations, the creation of virtual users allows volume testing to occur without
the need to have a large number of dedicated personnel resources. For some companies, this means testing enterprise
applications with hundreds or thousands of desktop, terminal, Web and Java clients to see the effect of Java users
on a system when accessing an e-commerce application through the Web. In other cases, it means validating different
bandwidths to determine user response time through a 56K connection vs. an ISDN or T1 connection.
Test once, monitor everywhere!
Despite copious testing, unforeseen problems can still occur once a system is placed into the production environment
and made available to increasing thousands of users and customers. In most situations, these problems are typically
related to performance.
To stem these problems, some companies have implemented 'early warning' systems to continuously monitor key
transactions and business functions.
The monitoring systems provided by software vendors are typically extensions of existing load-testing tools.
They contain the ability to identify specific components within the application, and allow QA analysts or system
administrators to establish acceptable thresholds for response time
or transaction usage. In addition, these thresholds can be established to exclusively monitor transactions or business
functions during specific times of the day when load is expected to be heavy.
But, according to Salemme at MotherNature.com, although usage of these tools can prove beneficial, they must
be balanced with any known or perceived resource constraints within the environment. "You've got to watch
out for overhead and make sure the monitoring tool doesn't impact response time," he said.
Back to the front
The Internet has quickly become a business mechanism that places the corporation in direct contact with customers
and suppliers. Those leading corporations that are taking advantage of Internet technology are building a new "store-front"
where customers can browse inventory, purchase wares and secure product support 24 hours a day, seven days a week.
The increasingly complex Web and e-commerce applications require greater disciplines in the development and
testing of quality applications. And, according to Open Market's Troppito, automated testing is the only way to
successfully ensure that all portions of the application are accurately exercised. "Automation is so important,"
he said. "It provides a robust solution across so many different platforms and it is a lot cheaper than adding
personnel resources to validate the quality of an application."