Testing tool makers scramble to keep up

While some things have become obsolete as technology has matured, others have increased in value. Consider automated testing tools (see the table "Sampling of Testing Tools" at the end of this article). This market has been changing since it began. Automated testing tools replaced long, drawn-out manual testing processes. Then the market shifted to testing packaged apps, followed by the Y2K hype. Now the Internet is driving the market.

Demand for testing tools has increased as development times have decreased. Today's testing products must be more scalable than ever as the number of users accessing the Internet has increased. The biggest driving factor in the testing tools market today is a shorter time to market.

To remain competitive and stay in business, testing tools vendors need to anticipate technology changes before they happen. Today vendors are looking at the bigger picture of life cycle management and are developing tools that can be used in each stage of the life cycle. Development today is an ongoing process: The sooner a defect, problem or bug is identified, the cheaper it is to fix and the less rework is necessary to solve the problem.

This is not going to change soon. As technology continues to evolve, testing tools do the same to keep up with it. Worldwide revenue for the automated software quality tools market grew almost 24% between 1999 and 2000 to nearly $1.2 billion, market research firm International Data Corp. (IDC), Framingham, Mass., reported. IDC forecasted that worldwide revenue will exceed $3 billion by 2005 with a compound annual growth rate of 21.4%.

With the Internet driving testing efforts, load and performance testing tools have gained more attention. Software developers have come to rely on tools like LoadRunner from Mercury Interactive, QACenter from Compuware, Rational QualityArchitect from Rational Software, SilkPerformer from Segue Software and WebLoad from RadView Software, among others, for testing how much load Web sites can handle and how they perform under stress.

Regression tools like Mercury WinRunner and Segue SilkTest help ensure that quality has not declined during development, and are also still common. Some vendors offer testing suites that cover both load/performance and regression testing, as well as other areas. These include e-Test Suite from Empirix Inc. (formerly RSW), Waltham, Mass., and Rational Suite TestStudio.

As e-businesses are beginning to realize the inherent value in offering Web services, most testing tools vendors are doing the same and offering hosted Web-based testing services in addition to their software products. "One of the biggest trends we're seeing is the appearance of hosted development services," noted Theresa Lanowitz, research director for Gartner Inc. in San Jose, Calif. "Load testing seems to be a natural fit for these hosted services."

Mercury was one of the first companies to offer testing services with its ActiveTest hosted load-testing service and ActiveWatch hosted Web performance monitoring service. Other companies have since followed suit. Keynote (formerly Envive Corp.), Mountain View, Calif., offers load testing and service-level monitoring services. Compuware offers a testing and monitoring service in PointForward. Segue provides a scalability testing service through SilkExpress. Rational recently partnered with third-party testing companies to offer testing services through its Rational Test Bureau program.

Hosted services are attractive to companies that do not have a performance or load testing expert on staff. In addition, these services eliminate the need to learn to use a new tool and to understand how to interpret test results. They also allow companies to leverage testing technology without having to buy the testing tools and bring them on-site. By providing a testing environment as well as expertise, services are saving companies time and gaining in popularity.

Jeffrey Peter, director of quality assurance for Covisint LLC, a Southfield, Mich., business-to-business exchange for the automotive industry, has used both testing services and software. Covisint develops some applications and also works as a systems integrator. Using a service, Peter said, makes an easier entry point for people who need to start performance monitoring and testing very quickly. Because Covisint is into long-term testing, however, Peter prefers to own the testing tools so he and his company can work on them themselves. The company currently uses Mercury TestDirector, LoadRunner, WinRunner and Topaz.

However, Dick Heiman, research director for application development and deployment at IDC, believes the testing services market will gain more traction. "These complex Web environments take a certain set of skills to write test scripts," he said. "Those skills are hard to find."

In addition, Web services themselves add a level of complexity to the mix. "Complexity just went up by tying Web applications together from different sources over the Internet and stitching them together in real time through Web services," explained Jayaram Bhat, Mercury's VP of marketing.

Hello, centralized testing!

Is there a middle ground between rolling your own tests and outsourcing tests entirely?

There may be for large organizations, said representatives of Sunnyvale, Calif.-based Mercury Interactive Corp., who suggested that the testing needs of dispersed development teams can be met by Web-enabled testing tools employed by a central test group.

Mercury recently released LoadRunner TestCenter to allow big companies to manage multiple load testing projects across various locations. The product is said to control all aspects of large-scale load testing projects, including resource allocation and scheduling, from a centralized location accessible via the Web.

This allows the cost of tools to be shared across divisions and projects. People within the organization can also gain testing expertise, ensuring that the organization does not need to "reinvent the wheel" to create test cases or solve performance problems that tests uncover.

"What we're seeing in large organizations is the centralization of test, and the Web is part of that," said Jonathan Rende, vice president of test product marketing, Mercury.

Rende outlined some of the drivers behind the trend. "People can gain expertise. The more you load test, the better you get at it. You start to know where the problems are. If you do it only once a year, you're not going to be nearly as effective," he said.

What may drive a large organization to centralize testing may drive a small firm to outsource to the growing legion of test services. The increasingly complex load test infrastructure needed for testing Web apps can be pricey.

As Melinda-Carol Ballou, senior research analyst at Westborough, Mass.-based MetaGroup, said, "It depends on how much you're going to use it. These tools are expensive to purchase and set up. There can be a lot of infrastructure required in hardware. But if it is going to see intensive use over an extended period of time, you may want to bring it in-house."

—Jack Vaughan

Shifting market
Hosted services is one of many trends in the testing tools market. There is also a shift toward integration. Although testing and monitoring were separate aspects of development, today they are overlapping. "Testing was a development function," IDC's Heiman said. "Monitoring was more at the service level, in the world of the operations management folks. It's a closed-loop system now where you do testing and also do monitoring often as a service so people can do wellness checks."

Cathy Lippert, director of product management for distributed products at Compuware has been noticing the same thing. "Just as testing is being applied earlier and earlier in the life cycle, we're also seeing interest in doing some synthetic transactions within operations," she said.

Remote testing is also gaining in popularity. "As you get more and more distributed applications, the success and delivery of any business application involves the ability to manage and use applications and databases that go past the firewall and into open space," explained Rich Ptak, senior vice president of Hurwitz Group Inc., Framingham, Mass.

Compuware's PointForward monitoring service works remotely. Compuware has points of presence around the globe from which it monitors applications. PointForward provides reliability, integrity and scalability testing, as well as performance monitoring. "Monitoring or testing remotely from different points of presence provides some inherent value with an external server," the company's Lippert said.

"Some of our customers try to do some stuff internally and find a different perspective once they move it outside the firewall," added Mike Killian, product manager for PointForward. In addition, he said, people connect to the Internet in different ways. "To test completely, you need to take those things into consideration."

That is why Countrywide Home Loans, a top U.S. mortgage lender based in Calabasas, Calif., tests remotely from an offshore facility in India. With 65 Web sites, 500 branch offices and a huge network infrastructure to maintain, testing is important for Countrywide. The company relies on e-Test Suite from Empirix for regression and load testing of its various sites. It tests each of its primary sites on a weekly basis.

During the development process, a developer can use e-Test Suite to develop a test script for a specific component and then stress test it to see how many users it can handle. "As an application is developed, the test script grows and comes right along with the application when it's released and can be used in monitoring applications and even in remote locations," explained Larry Gentry, Countrywide's first vice president of e-business technology.

The value of Countrywide's Web sites lies in the applications rather than in the static content, Gentry said. Thorough testing ensures this value. "Making all that work and keeping it available 24x7 at a satisfactory performance level is quite a challenge. Without a good testing and monitoring system, you almost can't do it. It becomes so resource-intensive that it's almost impossible," he added.

Another area that is gaining attention is testing for wireless and embedded systems. "Most, if not all, major testing vendors have announced support for some sort of wireless testing," Gartner's Lanowitz noted. She said she expects some expansion in this area during the next 12 to 18 months.

One vendor is meeting this challenge head-on. After concentrating on consulting for more than a decade, Minneapolis-based TestQuest introduced a product about two and a half years ago to test wireless handheld devices (including PDAs and cell phones), interactive television and general computer systems.

TestQuest Pro provides non-intrusive, whole-product functional testing, meaning it tests the behavior of an application while simultaneously testing how that application works with an entire system. The combined software and hardware solution simulates a user's experience from turning on a Palm device to bringing up an application to test. "A virtual user is actually pressing buttons, touching the touch panel, creating graffiti and behaving just like a user," explained Joe Toste, TestQuest's vice president of marketing.

Larry Lang, director of product quality at Handspring, Mountain View, Calif., found TestQuest Pro to be "the only game in town where we can actually test ROM." Handspring received a Palm version of the product in January and a Handspring version in late June. TestQuest integrated a Handspring computing device into its testing environment.

Traditional testing tools vendors are taking notice as well, as Lanowitz reported. The key to successful wireless testing is to be able to test at the component level, before a product is complete. Component testing also lets developers test earlier and more often in the development life cycle. Mercury offers support for wireless testing in LoadRunner and ActiveTest. Rational offers Rational Test RealTime to test wireless and embedded systems. Other vendors offer support as well.

ThinAirApps Inc., a New York wireless software infrastructure company, is using Rational Suite TestStudio to develop infrastructures that let enterprises extend their applications to wireless handheld devices. ThinAirApps' software runs on multiple devices, so thorough testing is mandatory.

"For any given feature, there are dozens of different combinations because of all the devices we can plug into," explained Evan Simeone, senior product manager. He likened it to testing a Web site to work in both Microsoft Explorer and Netscape Navigator - times one thousand. "Without a comprehensive automated testing plan, it's nearly impossible for us to release [a product] in a timely fashion. There are just too many variables in our business. We'd spend forever hunting them down."

Testing is really about risk management. "We help businesses minimize risks associated with development in accelerated development life cycles," Covisint's Peter noted. "It's critical to test processes in place that allow you to put out quality products that are going to help someone do their business rather than have the opposite effect."

An eye to the future
The future of testing is certain: There is one. As TestQuest's Toste noted, "In software development on small devices, reality is these devices are getting smaller but the software is getting considerably more complex. It's becoming key for vendors to run through regression testing." As software continues to increase in complexity, testing will continue to gain in importance and value.

One of the real challenges of testing is creating reusable test scripts. Rather than simply capture and play back a test sequence to reuse, however, a test case needs to be created at a slightly higher level to be efficient. Otherwise, as soon as the user interface changes, the test case is obsolete. "The test case needs to be created at a higher level so if the underlying infrastructure is there, you can use it with other things," noted Handspring's Lang. Testing tools vendors continue to address this issue.

As time moves on, more testing services and more combinations of monitoring and testing can be expected. Mercury, for example, can customize its testing offerings to customers because it offers software, hosted services and software running on a hosted service. The various combinations of these offerings provide customers with numerous testing options.

The market will continue to move toward integration. ThinAirApps, for instance, is already looking to unify its entire testing process into an integrated suite. Other companies are interested in doing the same. Testing tools vendors will have to address that unification.

Another thing to watch for is support for different types of media, such as video and audio streaming media. This support is already in progress. By teaming RSW Software with Hammer Technologies, Empirix is able to provide call center testing. The company's Multimedia Test Ensemble lets organizations test customer-facing voice and Web applications and infrastructure. Similar offerings are sure to follow.

As the market continues to mature, testing will become more and more critical. When the Internet first dawned, companies could get away with a lack of performance and occasional downtime. "If you're seriously playing in the e-business space right now, you simply can't do that anymore," Countrywide's Gentry said. "Applications have to be robust and scalable. Performance has to be there."

Sampling of testing tools
Company Product Features Test support Test environments Price
Atesto Technologies Inc.
Fremont, Calif.
Atesto Function Checker Automated script generation object-level tracking, checkpointing Function, regression Web applications 3-user license starts at $399/month
Atesto LoadModeler Multilocation end-user modeling, data-driven end-to-end performance testing Performance Internet and Intranet applications $1,500 to $20,000/month
AutoTester ONE Automated testing, task automation Load/performance, stress, function, regression, task automation, integration Web, Windows, SAP R/3, host/legacy, mainframe applications Starts at $3,000
BetaSphere Inc.
Pala Alto, Calif.
Feedback Management Server (FMS) Issue tracker, reporting feedback center, recruitment to-do lists, evalutions Usability function performance through collaboration N/A-automates collecting, managing acting on feedback Starts at $100,000
Candle Corp.
El Segundo, Calif.
eBusiness Assurance Network (eBAN) service Objective service-level metrics, supports multiple environments and transactional SLAs Measurement analysis Service-Level Agreement (SLAs) Starts at $2,500/month
Compuware Corp.
Farmington Hills, Mich.
PointForward Tells when an application problem exists, why and how to fix it. Integrity, reliability scalability, performance monitoring All Web environments Starts at $200
QACenter Performance Edition Record/playback virtual user transactions, creates baseline test data and test transaction variable data Load/performance, test data management server monitoring E-commerce, wireless ERP/CRM, distributed client/server applications Starts at $35,000
DSP Research Inc.
Sunnyvale, Calif.
FleXDS PCI-based in-circuit emulator, JTAG POD with 10-foot detachable cable Function Any application software $2,995
Empirix Inc.
Waltham, Mass.
e-Test Suite Visual Script technology server performance monitors, detailed diagnostics and reports Load/performance, function, regression monitoring Web-based applications Starts at $10,000
FirstACT Flexible reporting, automatic alerts Load, function Web services, COM components Starts at $20,995
eValid Inc.
San Francisco
eValid Advanced record/playback, metrics pop-up, dynamic link checking, extended reporting logs Load/performance, regression, function, content validation Web sites on Windows NT/2000 Starts at $950
Flashline QA Lab Testing results analysis, customized reports, three testing levels Code structure, analysis, load, performance, compatibility Java components $1,500 trial pack of 10 tests
InCert Software
Cambridge, Mass.
TraceBack Root cause diagnosis, monitoring, early warning Performance, function Windows C/C++, Visual Basic, Java, Solaris, OS/390 $495/seat
McCabe & Associates Inc.
Columbia, Md.
McCabe Test Instrumentation, source code preparation, command line interface planning Unit, integration, code coverage C, C++, Cobol, Visual Basic, Java, Fortran, Ada $30,000 to $60,000/5 concurrent licenses
Mercury Interactive Corp.
Sunnyvale, Calif.
LoadRunner Generates thousands of virtual users, real-time performance monitors, WebTrace analysis Load/performance Web and non-Web enterprise applications Starts at $40,000
WinRunner Intuitive recording process, checkpoints, interactive reporting tools, GUI map Function, regression Web browsers, Java apps, ActiveX, wireless, ERP/CRM, terminal emulators, VB, C/C++, PowerBuilder, Delphi Starts at $4,995
NetMechanic Inc.
Huntsville, Ala.
Browser Photo One-click testing, online viewing Function, integrity Web applications $120 annual subscription
HTML Toolbox Whole-site testing, reporting, custom-configured tests, one-click code correction Compatibility, load link check Web applications Starts at $40/year for 6-100 pages
ParaSoft Corp.
Monrovia, Calif.
Jtest Automates white-box testing, enforces Java Construction, function, regression, static analysis Java environments Starts at $3,495
WebKing Rule wizard, automatic path creation, servlet check, tests back-end components Load, construction, function, regression, unit All n-tier Web applications Starts at $3,995
RadView Software Ltd.
Burlington, Mass
WebFT Open standards scripting language, automated script adaptation, integrated testing environment Function Web applications Starts at $4,995
WebLOAD Load profiler, real-time dashboard, performance boost, predefined report portfolios Load/performance Windows, Unix, Java, XML, ActiveX, WAP, HTTP, SNMP, streaming technologies Starts at $11,995/100 virtual clients
Rational Software Corp.
Cupertino, Calif.
Rational Quality Architect Interactive development, test scenarios before completion, progress assessment Function, performance EJBs, COM components $3,495 for Rational Rose Enterprise
Rational Suite TestStudio Open APIs, integrates and automates white- and black-box testing, reuses test scripts Reliability, function, regression, unit, performance Windows, Web apps, ActiveX, Oracle PowerBuilder, SAP $5,495
Reasoning Inc.
Mountain View, Calif.
Reasoning Automated Software Inspection Services Detailed defect report, management reports, can be used before unit testing without test cases Function, memory leaks C/C++ source code Starts at 10 to 20 cents/line of code
Response Networks
North Andover, Mass.
Pulsar xSP Performance monitoring at end-user locations, problem isolation, potential brownouts validation Performance Win32, Solaris, Linux, Web apps $50,000 to $100,000
Segue Software Inc.
Lexington, Mass.
SilkPerformer Support for wireless apps, automated session handling, project-based script editor Load/performance Web applications, components, Java, VB, TCP/IP Starts at $50,000/100 virtual users
SilkTest Unattended testing, built-in recovery system, LinkTester, single-recording testing Function, regression Web applications $6,495
Spirent Communications
SmartBits Division
Calabasas, Calif.
WebSuite Generates thousands of HTTP transactions/second, scales test bed Load/performance Web sites, firewalls $4,995
Starbase Corp.
Santa Ana, Calif.
CaliberRBT Cause-effect graphing, detailed natural language test cases, minimum tests, coverage analysis Function Any application Starts at $3,495/seat
TestQuest Inc.
TestQuest Pro Connectivity, test development environment, test manager application Function Computers, PDAs, cell phones, interactive TV Starts at $25,000
Santa Clara, Calif.
Quality Analyzer Open APIU, source code quality checker, Java code coverage, metrics Performance, integrity Java code $1,995
Source: Lana Gates