In-Depth

Testing e-commerce

• By Philip E. Courtney
• June 5, 2001

For IT managers, testing was never easy. But in the age of the Internet, it just got harder. Once you had your hands full testing for all the possible things that could happen within your company. Now your "users" come from all parts of the world, and they want their systems to work as easily and reliably as the family television. Suddenly, for IT, the world is like a living room. And things had better work.

But making things work in such a complex setting is daunting. "There are so many intertwined and interconnected items within a Web-based or e-commerce system that it's impossible to say any one discipline is less important than another," said David Richardson, chief technology officer at Rhino Productions Inc., a Tempe, Ariz.-based firm that develops custom e-commerce and Web applications. "Testing and QA disciplines must focus on the individual components, the entire infrastructure and how the system is used by the customer."

This is because most Web sites contain an underlying collection of diverse technologies that have been integrated to meet a variety of business requirements. For example, Acton, Mass.-based MotherNature.com is an e-commerce company that sells more than 30,000 natural health-care products in 1,400 different categories. The company uses a combination of tools found in the e-Test product suite from Watertown, Mass.-based RSW Software Inc. to ensure the accuracy of systems distributed across a multitude of daisy-chained servers and connected by a variety of networking technologies. MotherNature.com has assembled quality assurance and testing processes for individual components within the application, as well as load testing
for approximately 1,250 simultaneous, simulated customers.

Following the evolution of the Web from simple, online marketing brochures to full-fledged centers of e-commerce -- the Internet commerce application market will soar to $1.7 billion this year according to International Data Corp., Framingham, Mass. -- are dozens of testing tools designed to help ensure the quality of Internet applications. Although these products support a variety of Internet testing disciples, companies that have made heavy investments in Web-based applications state that the most important tools in their arsenal are those designed to assist with application, component and load testing.

In a variety of very public failures, a number of high-profile e-commerce sites have shown what can go wrong when systems are changed and testing takes a back seat. These development issues are, naturally, just as pressing at sites like MotherNature.com.

"Each product in our system has its own Web page connected to a data bank containing the unique ID and UPC code," said Vic Salemme, director of quality assurance at MotherNature.com. "We need to test customer access to each page and validate the ability to link products, codes and prices on an individual basis. We also need to ensure quality performance and eliminate bottlenecks during expected increases in volume."

The Web was once the exclusive domain of marketing. In recent years, the thrust toward e-commerce has served
to meld marketing and development requirements. This has helped instill traditional system development life-cycle methodologies for the creation of Web applications.

For testing, however, the question arises as to when disciplines and tools should begin to be used within the process. Without variation, the answer is once the requirements have been defined.

For instance, new Web development activities typically require two primary tasks. The first, driven by marketing types, is to build the look and feel -- or storyboard -- requirements. The other task, which is more development-oriented, is to build the internal design requirements. The completion of these tasks can produce a prototype that models the look and feel of the site from a user perspective, as well as the flow of interaction using the underlying technologies.

"You've got to try to get the process down first before applying tools," explained Pete LeFebvre, vice president of development at Gallagher Financial Systems, Coral Cables, Fla., a provider of mortgage origination and pipeline management software. "Applying tools too early during development activities can be counterproductive. Instead, the tools should be applied where they fit within the process."

Added Rhino's Richardson, "Test scripts and scenarios can be created once issues such as presentation, navigation, data design, data transfer and so on, have been resolved." And creating the scripts and scenarios is where the tools come into play.

Script building

Creation of test scripts for Web applications typically occurs as development is nearing completion and takes the form of regression testing. Nearly identical to the disciplines used for validating the GUI screens of client/server applications, regression testing for a Web-based system is one of the most fruitful methods for pinpointing errors as early as possible prior to a production implementation.

The caveat, however, is that the capture/playback tools used for regression testing must be capable of recognizing the differences between a traditional client/server application and a Web application.

One major difference is that, while standardized screens are common in the client/server environment, a user can access a Web application through any number of different browsers running on any number of different operating systems. Today, for example, both Microsoft and Netscape each have at least three predominant versions of their browsers on the market -- and what works in one may not necessarily work in another.

"Internet Explorer 3.0 doesn't work very well with Active Server Pages [ASPs] or Java, but we have few problems with IE 4.0 or IE 5.0," said Kathy Kirk, QA lead at custom software and e-commerce development firm Key Tech Inc., Eagan, Minn. "We need to be able to test for a multitude of browsers and gain an understanding of what works and what doesn't work." To support its different application and load testing requirements, Key Tech uses Web- Tester from Cyrano, Newburyport, Mass.

In addition to browser differences, the look and feel of today's Web screens are always subject to change even though functionality remains the same. Using Active Server Pages, Java applets, ActiveX and other components to produce date/time stamps, banner pages, animation, tickers and other dynamic effects, a site accessed by a user in the morning can appear dramatically different by afternoon -- without a change in functionality. Accurate testing dictates that the capture/playback tools used for Web testing recognize and accommodate changes in appearance while also recognizing and pinpointing changes in functionality.

"You can't build a Web regression test based on what's on a screen," said Michael Maggio, senior vice president and general manager at Segue Software, Lexington, Mass. "Compared to GUI testing, the screens and the controls are very different."

Just like in GUI testing, the capture function in a regression-testing tool is used to automatically generate the scripts that will be used for playback. And while these scripts will be used for application testing, it is equally important that they remain reusable and flexible so that they can be employed again later in the cycle during the various scenarios established for load and performance testing.

"The scripts created early in the process must be reusable when the time comes to perform simulated load testing," said Salemme at MotherNature.com. "[This] avoids rework and allows different scripts to be used at different times by different virtual users when validating system performance."

Lastly, though, it is important that scripts provide support for one of the most prevalent challenges in testing: test data generation. As Salemme points out, "Our largest challenge was data banking all of our products so that they could be read by our scripts."

Support for test data comes with the ability to separate captured data, such as user entry and tables, from the scripts and place it into a file. During playback, the script will read the data from the created file -- or any other file. This functionality allows testers and QA analysts to alter test cases without altering a script.

"The best data comes from the real application," said Yves de Montcheuil, director of Web solutions at Cyrano. "For new test data generation some of the tools most frequently used are TestBytes [from Platinum Technologies Inc.] and DataShark [from HardBall Software Inc.]."

Component testing

But what happens when an application component fails during a regression test? In addition to basic HTML, today's Web-based applications comprise numerous controls, Java applets, Perl scripts and more. Underlying languages can be anything from C/C++ to Visual Basic.

While many problems in e-commerce and Web-based applications can be traced to performance issues, other failures are sometimes traced to problems within the underlying code -- code that must be analyzed and debugged. And in many cases, that code was developed on one virtual machine for use on many others that use different run-time characteristics. Other problems can be traced to memory leaks, stack errors or incorrect usage of APIs and links -- or just plain old syntax errors in the employed programming language.

But one of the biggest challenges at the component level is the variety of languages, platforms and environments used for development. While Visual Basic development typically occurs on the client-side of the house, Java developers typically focus on the server, and C++ can be found anywhere and everywhere; this joining of so many different individual components can wreak havoc when the sum of the parts become whole as a Web application.

"Without care in development and testing, the instantaneous worldwide deployment of an application can also mean the instantaneous worldwide deployment of bugs," said Doug Carrier, product manager for Java development tools at Compuware Corp.'s NuMega Labs, Nashua, N.H. "Problems typically occur when moving to a three-tier environment from a two-tier and the multiple components increase testing complexity."

While there a number of tools available to assist in pinpointing these problems, Rhino Productions' Richardson points out that developers must exercise a great deal of care when performing modifications at such a low level within the application. "The components and the programming languages are the lowest common denominator in most of these cases," he said. "Although it's important to make sure the problem is corrected, it's also important to test not only the single component, but the remainder of the application. The integrated infrastructure of the Web means that the one changed component can be used in any number of other locations within the application."

Others, however, have seen little need for the tools used to assist in debugging. They profess that most errors occur during times of high traffic, and that a proper design helps avoid errors at the lowest level.

"There are so many different pieces in a system," said Kirk at Key Tech Inc., "that there's really no justification in performing a line-by-line analysis of the code. We have enough trapped code available to perform a review of a problem if needed, but we've found that most problems occur in the infrastructure of the system."

Validating the infrastructure

Of all the disciplines invoked for validating Web-based applications, load testing has to be listed as the most crucial. As an interactive medium in a society grown accustomed to instant gratification, users and customers expect an almost immediate response when doing business over the Web. Patience is not a virtue, and a slow site results in lost business.

"Customers want and expect a reasonable response time," said Kirk. "They're most interested in load testing."

But load testing is almost like 'black magic' in terms of Web testing because it requires companies to identify both real and theoretical limits to create the proper scenarios. And the Web is a fast moving target. Still, this discipline helps companies identify, isolate and analyze the items that can have a detrimental effect on response time and performance, including configurations for networks and servers, communications and bandwidth, or just plain old volume.

When performed early enough in the development process, load testing allows organizations to remedy internal configuration issues prior to a production roll-out.

"We identified a number of configuration and resource issues during one early load test when we hit 30 concurrent users and the system just quit," explained LeFebvre at Gallagher Financial Systems. For load testing, his company employs LoadRunner from Mercury Interactive, Sunnyvale, Calif. Last month, Mercury released Version 6.0 of that product.

"Load testing is a matter of creating different scenarios to ensure proper performance," said Tony Troppito, director of software quality assurance at Burlington, Mass.-based Open Market, a company that produces software for commerce service providers. Open Market uses products from Segue's LiveQuality suite. "You've got to test for different configurations and different volumes of users. You've got to review specific situations such as 'How will this system handle 10,000 transactions per hour while another system is handling 5,000?'"

An important function needed in a load-testing tool is the ability to create scenarios as well as virtual users. While scenario creation allows QA analysts to establish different testing situations for a multitude of hardware, network and communications configurations, the creation of virtual users allows volume testing to occur without the need to have a large number of dedicated personnel resources. For some companies, this means testing enterprise applications with hundreds or thousands of desktop, terminal, Web and Java clients to see the effect of Java users on a system when accessing an e-commerce application through the Web. In other cases, it means validating different bandwidths to determine user response time through a 56K connection vs. an ISDN or T1 connection.

Test once, monitor everywhere!

Despite copious testing, unforeseen problems can still occur once a system is placed into the production environment and made available to increasing thousands of users and customers. In most situations, these problems are typically related to performance.

To stem these problems, some companies have implemented 'early warning' systems to continuously monitor key transactions and business functions.

The monitoring systems provided by software vendors are typically extensions of existing load-testing tools. They contain the ability to identify specific components within the application, and allow QA analysts or system administrators to establish acceptable thresholds for response time
or transaction usage. In addition, these thresholds can be established to exclusively monitor transactions or business functions during specific times of the day when load is expected to be heavy.

But, according to Salemme at MotherNature.com, although usage of these tools can prove beneficial, they must be balanced with any known or perceived resource constraints within the environment. "You've got to watch out for overhead and make sure the monitoring tool doesn't impact response time," he said.

Back to the front

The Internet has quickly become a business mechanism that places the corporation in direct contact with customers and suppliers. Those leading corporations that are taking advantage of Internet technology are building a new "store-front" where customers can browse inventory, purchase wares and secure product support 24 hours a day, seven days a week.

The increasingly complex Web and e-commerce applications require greater disciplines in the development and testing of quality applications. And, according to Open Market's Troppito, automated testing is the only way to successfully ensure that all portions of the application are accurately exercised. "Automation is so important," he said. "It provides a robust solution across so many different platforms and it is a lot cheaper than adding personnel resources to validate the quality of an application."