In-Depth

Are automated test tools for real?

• By Alan R. Earls
• May 1, 2004

While automating the testing process has the potential to raise quality and stretch resources, automation is not magic. Industry watchers and even vendors agree that not all tests need to be automated, and that automation is not a panacea. Few applications can be completely tested through automation, and some say the biggest challenge is deciding which functions or controls should be automated. Skeptics point out that demos of automated testing tools look impressive, but the tools too often become "shelfware." While some observers suggest that people should build their own testing tools, supporters of automated testing say the tools have improved in recent years and that long-term trends favor automation. Many users, especially those who do repetitive tests throughout the application life cycle, are enthusiastic about their automated testing tools.

"I think the use of the word 'phenomenon' seriously understates the value of automated testing," said Dan Young, lead automation engineer in the Performance Technologies Group at Charles Schwab, one of the nation's largest financial services firms. Young said his company uses automation -- specifically Segue SilkTest -- to run nearly 7,000 test cases per night. "It would take a manual tester roughly four months of work to produce the results that we produce every single night with automated testing," he said. Young said issue discovery is passed to the development team daily; when a code fix is implemented, it is tested that night and the results are reported the following day.

Automated testing is far from being a fad, said Young; in fact, he added, "it has become an institution in the software industry." A lot of companies that were content to rely on their customers to test their products no longer exist. "Any company that is serious about the quality of its products recognizes the value of testing in-house," said Young. "It's an easy step from that point to seeing how automated testing can be used to leverage resources and keep testing costs down in the long run."

While Young's enthusiasm reflects the genuine capabilities of some automated test products -- and a successful deployment -- others in the industry caution that automation is not always so simple. They warn that it must be applied appropriately with an understanding of the problems it seeks to address and with staff capable of using it wisely. Indeed, in a 2002 research report, Stamford, Conn.-based analyst firm Gartner Inc. noted that "not every test needs to be automated," particularly those that are run infrequently. Still, Gartner concluded, "when using automated tools ... it's easier to test more frequently," which fulfills the "test early and often" maxim of testing gurus who say it is between 10 and 10,000 times cheaper to remove a defect as early in the process as possible.

Automation's downside
Among those with a jaded view of most testing automation is Kanglin Li, who has worked as a software design engineer at Agilent Technologies, Palo Alto, Calif., and at Communications Data Services, Des Moines, Iowa, and has served as assistant professor at North Carolina A&T University in Greensboro, N.C. With his wife, Mengqi Wu, a system engineer at Lucent Technologies' Bell Labs in Murray Hill, N.J., Li recently wrote a book entitled Effective Software Test Automation: Developing an Automated Software Testing Tool (Sybex Inc., Alameda, Calif., 2004) in which he is sharply critical of commercially available automated test tools. Indeed, Li recommends that customers simply build their own testing tools.

According to Li, test engineers he has spoken with report that commercial testing tools are very impressive when the vendors are conducting demonstrations. But once the testers purchase the tools and learn to use them, however, they become "shelfware."

"These tools can automatically generate test scripts to do some basic things," admitted Li, "but when testers need to test software in depth, they have to add functions into the generated test scripts." And that can be a complex and time-consuming task.

"Testers desire to automate software testing as much as possible. But using the commercial tools is expensive. It will pay back the investment only if the automated test scripts are reused frequently," said Li.

Like Li, Gartner analyst Theresa Lanowitz offers strong words of caution about testing automation. Still, she said, "the tools themselves are very good and have matured a lot in the last few years -- some even provide best practices built-in."

The challenge, Lanowitz explained, is to provide what she calls the "three Ps": people, process and product. She said that if you are considering adopting automated tools, you need to make sure you have skilled professionals in place and a good process. "When we hear about failures it is easy to blame the tool because it is inanimate," she said. "In reality, most of the time the tool isn't to blame" -- usually what fails is the combination of the people, process and product, she said. Too often, people do not have the proper training or are not given time to come up to speed with the tool, and there is usually no documented, repeatable development and test process. More broadly, she noted, "testing itself is seen as frivolous, so there isn't support from top management." And, she added, "people think if they aren't coding they aren't working."

Lanowitz said that given the amount of money lost due to poor software quality -- billions of dollars annually in the U.S. -- additional testing capabilities like automation should be a plus. "If an app goes down, on average it can cost a company $100,000 an hour if it is something that's mission-critical," she said.

Still, even with automation, it is up to people to do some fundamental tasks, including framing a solid requirements document. "If you look at the software development life cycle, most people think the bulk of their time should be spent in the construction phase, but requirements and design should get more of the attention -- that will make the construction phase shorter," Lanowitz said. If, in fact, you spend the bulk of the time on construction, you are guaranteed to spend even more on maintenance, she warned.

Lanowitz said greater automation, done on an ongoing basis, will significantly reduce risks. But, she added, that is assuming you have those "three Ps," and follow them version after version and app after app. "If you do that you will substantially improve your process and that will result in greater customer satisfaction," she said. And, she added, you will spend less time on maintenance by doing it right from the beginning.

"The old cliche that there is never enough time to do it right but always time to do it over, doesn't apply anymore because people don't have the luxury of time or money anymore," Lanowitz said. In other words, organizations must do it right the first time. "If they are doing things without proper quality, they are probably mistaking rework for the normal cost of developing an app," she added.

Automation is key, she said, "but it is not a silver bullet. Many organizations create an app, run into quality problems and buy a tool in the hope that it will be a panacea, but they have no repeatable process and no training for their people to think along these lines," Lanowitz explained.

Vendors' views vary
Even vendors admit that automation does not solve everything. "IT organizations have made significant investments in technology and tools -- but the industry as a whole has not seen the benefits that were expected," said Dave Kapelanski, technical director at Compuware Corp. in Detroit. He said automated testing has not progressed to the point where it provides the right information required to make informed go/no-go decisions.

"Every vendor has attempted to offer solutions -- from keyword-driven testing, to record and playback, to data-driven testing," he said. "But none of these technologies provides the insight to deploy applications."

In addition, he argued, automated test scripts account for only 25% of all test cases. According to Kapelanski, a more comprehensive testing approach must be utilized that ensures quality improvement throughout the entire application life cycle.

Kapelanski said Compuware believes that a systematic approach to "risk-based testing" is necessary to address the quality issues that cannot be resolved by point solutions during the test phase of the project. Adopting a risk-based methodology greatly improves the quality of applications, he said, allowing IT organizations in general, and the software quality function in particular, to confidently report on the readiness of an application for production.

Risk-based testing focuses on overall quality assurance not just testing, he explained. "Quality assurance and risk-based testing are about preventing problems from arising through requirements management and early delivery of test assets during the design and development process," Kapelanski said. On the other hand, "testing identifies what problems exist in the current application during the testing phase," he noted. Compuware's risk-based testing approach takes a holistic view of quality, identifying areas of business risk early and resolving them at the business requirements level.

"If you look at any of the automation tools, what they are is just programming languages that have a special ability to drive other applications," said Linda Hayes, CTO at Worksoft Inc. in Dallas. But the average person in QA is not a programmer -- instead they may have business expertise or they may be a quality analyst -- so there can be a big difference between the skills the people who actually use these tools have and those they would need to use them effectively.

Hayes charges that vendors habitually showcase demos that make automation look easy, but when people buy the tools, they find they just create a lot of bad code. "There is no error recovery or error handling," she said. That means the scripts are unstable, unreliable and not maintainable. "So if you don't have the skills to code, you are going to have a hard time fixing it," she said.

As a result, said Hayes, companies tend to make one of two decisions. They either put the product on the shelf or hire real coders, which means a development effort in the testing organization. That boils down to a "class system" she said, in which the people who know the most about the application are not the people doing the automation. "They are relegated to a subservient manual role while the experts do the automation," she said.

Worksoft, Hayes noted, offers a script-free product called Certify that lets analysts "point and click" to documents and automate their test cases -- eliminating recording, programming, testing, debugging and maintenance of scripts. "We enable the people who do manual testing to do automation," she said.

Paul Spicer, product manager at Candle Corp., El Segundo, Calif., [which IBM recently agreed to acquire] said his company focuses on early stage testing issues with a workbench product that focuses not so much on testing functionality as on "tuning and scalability issues" using a mixture of manual tools and service offerings. However, Spicer said Candle does see an important role for automation in stress testing, which is inherently oriented to repetition. Indeed, he said his company's product is often used in close conjunction with load testing tools.

Deborah Kablotsky, director of product management and support at RadView Software Ltd. in Burlington, Mass., offers a more robust defense of the automation capabilities available on the market. She said her company's product covers the spectrum of testing for Web-based applications from early in the definition phase through unit testing and even into post-production monitoring.

The manual process comes when you set up the test -- you want to try to create scenarios that are modular and repeatable, she said.

"It is a basic framework where people record how end users will interact with the application," she said. Kablotsky said RadView's products are designed to span the needs of QA professionals, from those who want to write their own code to those who simply want to point and click.

"It is up to you to customize, but the goal is to streamline the process of developing test scripts," she said.

With an even more pro-automation view, Jay Weiser, director of strategic solutions at Segue Software Inc. in Lexington, Mass., said long-term trends are on the side of automation. Automated software testing has been in existence since the early 1990s, he noted. He points to his own company's history in which many people and organizations have adopted its products, such as SilkTest and its predecessor QA Partner.

"The ability to build GUI tests that span builds as well as versions allows development groups to ensure that new code does not introduce new defects in previously functioning code, and to rerun these tests repeatedly over the lifetime of an application," he said.

Furthermore, test script functionality can be reused across applications in the same way that application code can be reused. For example, he said, a DLL can be developed to encapsulate common functions for use by more than one development effort. Similarly, a set of common script functionality can be written to handle a common login, for instance, which only needs to be written once but can be used for all applications that share a common corporate login. In addition, Weiser noted, automation ensures that tests are run the same way each time, avoiding inadvertent false defects caused by human input error -- "something that is especially likely in highly repetitive tests for which automated tests are very well suited," he said.

Weiser also noted that SilkTest includes features like an automated recovery system that resets an application under test prior to each test case. This allows tests to be run in a fully unattended, lights-out fashion during overnight hours or weekends, thus extending the testing hours, making maximum use of test resources and shortening QA cycles. SilkTest, Weiser commented, includes a distributed application allowing testing to be done across multi-step business processes or for testing a matrix of operating systems or environments.

There are very few applications that can be tested 100% via automation, Weiser admitted. In fact, he said, 75% to 80% is a pretty high degree of automation. In his view, some applications can be automated to well over 90%, while other applications cannot even be 50% automated. A large part of automating a testing process is in determining which functions or controls can or should be automated.

In general, true quality automation is not used as much to find bugs in new code as it is to ensure that defects have not been introduced in previously functioning parts of the application, he said. Writing or modifying scripts to test new functionality generally uncovers defects in new code, while existing automation is an insurance policy for development groups that protects an existing code base.

Despite the skeptics, it is clear that many customers are buying the automation argument.

Schwab's Young, for one, said Segue SilkTest has proven to be very reliable. "Given the relatively open structure of SilkTest and the powerful development language it provides, we are able to run our nightly regressions with nearly a 100% success rate," he said.

Failures almost never occur because of the automation tool, he said, but rather are caused by changing program code, automation code or differences in how operating systems recognize particular window objects.

Please see the following related story: "A sampling of automated testing tools" by Lana Gates