In-Depth

Testing tool makers scramble to keep up

• By Lana Gates
• October 1, 2001

Demand for testing tools has increased as development times have decreased. Today's testing products must be more scalable than ever as the number of users accessing the Internet has increased. The biggest driving factor in the testing tools market today is a shorter time to market.

To remain competitive and stay in business, testing tools vendors need to anticipate technology changes before they happen. Today vendors are looking at the bigger picture of life cycle management and are developing tools that can be used in each stage of the life cycle. Development today is an ongoing process: The sooner a defect, problem or bug is identified, the cheaper it is to fix and the less rework is necessary to solve the problem.

This is not going to change soon. As technology continues to evolve, testing tools do the same to keep up with it. Worldwide revenue for the automated software quality tools market grew almost 24% between 1999 and 2000 to nearly $1.2 billion, market research firm International Data Corp. (IDC), Framingham, Mass., reported. IDC forecasted that worldwide revenue will exceed $3 billion by 2005 with a compound annual growth rate of 21.4%.

With the Internet driving testing efforts, load and performance testing tools have gained more attention. Software developers have come to rely on tools like LoadRunner from Mercury Interactive, QACenter from Compuware, Rational QualityArchitect from Rational Software, SilkPerformer from Segue Software and WebLoad from RadView Software, among others, for testing how much load Web sites can handle and how they perform under stress.

Regression tools like Mercury WinRunner and Segue SilkTest help ensure that quality has not declined during development, and are also still common. Some vendors offer testing suites that cover both load/performance and regression testing, as well as other areas. These include e-Test Suite from Empirix Inc. (formerly RSW), Waltham, Mass., and Rational Suite TestStudio.

As e-businesses are beginning to realize the inherent value in offering Web services, most testing tools vendors are doing the same and offering hosted Web-based testing services in addition to their software products. "One of the biggest trends we're seeing is the appearance of hosted development services," noted Theresa Lanowitz, research director for Gartner Inc. in San Jose, Calif. "Load testing seems to be a natural fit for these hosted services."

Mercury was one of the first companies to offer testing services with its ActiveTest hosted load-testing service and ActiveWatch hosted Web performance monitoring service. Other companies have since followed suit. Keynote (formerly Envive Corp.), Mountain View, Calif., offers load testing and service-level monitoring services. Compuware offers a testing and monitoring service in PointForward. Segue provides a scalability testing service through SilkExpress. Rational recently partnered with third-party testing companies to offer testing services through its Rational Test Bureau program.

Hosted services are attractive to companies that do not have a performance or load testing expert on staff. In addition, these services eliminate the need to learn to use a new tool and to understand how to interpret test results. They also allow companies to leverage testing technology without having to buy the testing tools and bring them on-site. By providing a testing environment as well as expertise, services are saving companies time and gaining in popularity.

Jeffrey Peter, director of quality assurance for Covisint LLC, a Southfield, Mich., business-to-business exchange for the automotive industry, has used both testing services and software. Covisint develops some applications and also works as a systems integrator. Using a service, Peter said, makes an easier entry point for people who need to start performance monitoring and testing very quickly. Because Covisint is into long-term testing, however, Peter prefers to own the testing tools so he and his company can work on them themselves. The company currently uses Mercury TestDirector, LoadRunner, WinRunner and Topaz.

However, Dick Heiman, research director for application development and deployment at IDC, believes the testing services market will gain more traction. "These complex Web environments take a certain set of skills to write test scripts," he said. "Those skills are hard to find."

In addition, Web services themselves add a level of complexity to the mix. "Complexity just went up by tying Web applications together from different sources over the Internet and stitching them together in real time through Web services," explained Jayaram Bhat, Mercury's VP of marketing.

Shifting market
Hosted services is one of many trends in the testing tools market. There is also a shift toward integration. Although testing and monitoring were separate aspects of development, today they are overlapping. "Testing was a development function," IDC's Heiman said. "Monitoring was more at the service level, in the world of the operations management folks. It's a closed-loop system now where you do testing and also do monitoring often as a service so people can do wellness checks."

Cathy Lippert, director of product management for distributed products at Compuware has been noticing the same thing. "Just as testing is being applied earlier and earlier in the life cycle, we're also seeing interest in doing some synthetic transactions within operations," she said.

Remote testing is also gaining in popularity. "As you get more and more distributed applications, the success and delivery of any business application involves the ability to manage and use applications and databases that go past the firewall and into open space," explained Rich Ptak, senior vice president of Hurwitz Group Inc., Framingham, Mass.

Compuware's PointForward monitoring service works remotely. Compuware has points of presence around the globe from which it monitors applications. PointForward provides reliability, integrity and scalability testing, as well as performance monitoring. "Monitoring or testing remotely from different points of presence provides some inherent value with an external server," the company's Lippert said.

"Some of our customers try to do some stuff internally and find a different perspective once they move it outside the firewall," added Mike Killian, product manager for PointForward. In addition, he said, people connect to the Internet in different ways. "To test completely, you need to take those things into consideration."

That is why Countrywide Home Loans, a top U.S. mortgage lender based in Calabasas, Calif., tests remotely from an offshore facility in India. With 65 Web sites, 500 branch offices and a huge network infrastructure to maintain, testing is important for Countrywide. The company relies on e-Test Suite from Empirix for regression and load testing of its various sites. It tests each of its primary sites on a weekly basis.

During the development process, a developer can use e-Test Suite to develop a test script for a specific component and then stress test it to see how many users it can handle. "As an application is developed, the test script grows and comes right along with the application when it's released and can be used in monitoring applications and even in remote locations," explained Larry Gentry, Countrywide's first vice president of e-business technology.

The value of Countrywide's Web sites lies in the applications rather than in the static content, Gentry said. Thorough testing ensures this value. "Making all that work and keeping it available 24x7 at a satisfactory performance level is quite a challenge. Without a good testing and monitoring system, you almost can't do it. It becomes so resource-intensive that it's almost impossible," he added.

Another area that is gaining attention is testing for wireless and embedded systems. "Most, if not all, major testing vendors have announced support for some sort of wireless testing," Gartner's Lanowitz noted. She said she expects some expansion in this area during the next 12 to 18 months.

One vendor is meeting this challenge head-on. After concentrating on consulting for more than a decade, Minneapolis-based TestQuest introduced a product about two and a half years ago to test wireless handheld devices (including PDAs and cell phones), interactive television and general computer systems.

TestQuest Pro provides non-intrusive, whole-product functional testing, meaning it tests the behavior of an application while simultaneously testing how that application works with an entire system. The combined software and hardware solution simulates a user's experience from turning on a Palm device to bringing up an application to test. "A virtual user is actually pressing buttons, touching the touch panel, creating graffiti and behaving just like a user," explained Joe Toste, TestQuest's vice president of marketing.

Larry Lang, director of product quality at Handspring, Mountain View, Calif., found TestQuest Pro to be "the only game in town where we can actually test ROM." Handspring received a Palm version of the product in January and a Handspring version in late June. TestQuest integrated a Handspring computing device into its testing environment.

Traditional testing tools vendors are taking notice as well, as Lanowitz reported. The key to successful wireless testing is to be able to test at the component level, before a product is complete. Component testing also lets developers test earlier and more often in the development life cycle. Mercury offers support for wireless testing in LoadRunner and ActiveTest. Rational offers Rational Test RealTime to test wireless and embedded systems. Other vendors offer support as well.

ThinAirApps Inc., a New York wireless software infrastructure company, is using Rational Suite TestStudio to develop infrastructures that let enterprises extend their applications to wireless handheld devices. ThinAirApps' software runs on multiple devices, so thorough testing is mandatory.

"For any given feature, there are dozens of different combinations because of all the devices we can plug into," explained Evan Simeone, senior product manager. He likened it to testing a Web site to work in both Microsoft Explorer and Netscape Navigator - times one thousand. "Without a comprehensive automated testing plan, it's nearly impossible for us to release [a product] in a timely fashion. There are just too many variables in our business. We'd spend forever hunting them down."

Testing is really about risk management. "We help businesses minimize risks associated with development in accelerated development life cycles," Covisint's Peter noted. "It's critical to test processes in place that allow you to put out quality products that are going to help someone do their business rather than have the opposite effect."

An eye to the future
The future of testing is certain: There is one. As TestQuest's Toste noted, "In software development on small devices, reality is these devices are getting smaller but the software is getting considerably more complex. It's becoming key for vendors to run through regression testing." As software continues to increase in complexity, testing will continue to gain in importance and value.

One of the real challenges of testing is creating reusable test scripts. Rather than simply capture and play back a test sequence to reuse, however, a test case needs to be created at a slightly higher level to be efficient. Otherwise, as soon as the user interface changes, the test case is obsolete. "The test case needs to be created at a higher level so if the underlying infrastructure is there, you can use it with other things," noted Handspring's Lang. Testing tools vendors continue to address this issue.

As time moves on, more testing services and more combinations of monitoring and testing can be expected. Mercury, for example, can customize its testing offerings to customers because it offers software, hosted services and software running on a hosted service. The various combinations of these offerings provide customers with numerous testing options.

The market will continue to move toward integration. ThinAirApps, for instance, is already looking to unify its entire testing process into an integrated suite. Other companies are interested in doing the same. Testing tools vendors will have to address that unification.

Another thing to watch for is support for different types of media, such as video and audio streaming media. This support is already in progress. By teaming RSW Software with Hammer Technologies, Empirix is able to provide call center testing. The company's Multimedia Test Ensemble lets organizations test customer-facing voice and Web applications and infrastructure. Similar offerings are sure to follow.

As the market continues to mature, testing will become more and more critical. When the Internet first dawned, companies could get away with a lack of performance and occasional downtime. "If you're seriously playing in the e-business space right now, you simply can't do that anymore," Countrywide's Gentry said. "Applications have to be robust and scalable. Performance has to be there."