In-Depth

Quality quest

• By Billie Shea
• June 25, 2001

Since its emergence into its own technology discipline more than a decade ago, automated testing tool vendors have struggled to make their voice and message heard. Automated testing is gaining more attention as twin forces -- the need to grapple with year 2000 (Y2K) dates in legacy code, and the desire to do commerce on the World Wide Web -- confront application developers.

Consider the complexity involved in the deployment of transaction-based Web applications. Multiple software variables such as browser types, HTML, XML, JavaScript and Java applets, Microsoft Active Server Pages (ASPs), ActiveX controls, communication protocols, operating systems and back-end databases combine with various hardware platforms and network structures. These variables connected to Web application development and deployment all represent points of failure for continuously available Web applications. Strengthening the case for quality, Web applications represent the integrity of an organization's business operations and expose the organization to a large audience of external users (for example, customers and business partners). It is for these reasons that organizations seeking to absorb electronic commerce into their existing I/S structures have an increased re sponsibility to ensure the stability, reliability, transaction integrity and overall quality of the applications developed for conducting e-business.

On another front, the Y2K crisis has pushed organizations to examine and fine-tune their quality assurance (QA) practices and to define (and secure) specific budgets for them. For all of the expense and anxiety the Y2K crisis has caused, it has done well to inspire operational changes throughout many an I/S department. First, Y2K is classified as a business problem that requires a technical fix. It has forced senior executives to take a closer look at their company's technology underpinnings in order to justify budgets for Y2K compliance. Moreover, Y2K has forced I/S to inventory software, detail a solid plan of action and employ adequate testing.

Still, convincing Corporate America of the value in adopting automated tools and formal QA practices has been nothing short of an uphill climb. Conceptually, the quality messages and value to the business are compelling. I/S teams believe in enhancing the quality of their applications, but have often found that justifying the price of an automated tool and the cost of formalizing a practice (for example, employing a QA team to utilize the tool and manage the process) are obstacles. Furthermore, when applications are pushed for deployment, testing is a good place to trim when time runs short.

"Quality issues have definitely grown and [end-user] expectations keep getting higher and higher," said Herb Isenberg, quality assurance architect for Oacis Healthcare Systems, Greenbrae, Calif. Isenberg has been using automated tools from AutoTester Inc., Dallas, for the past decade and believes the Y2K issue has been a catalyst for the heightened focus on quality. "Executive management is cutting the big checks to resolve the compliance issue and they want to ensure that the issues are resolved, which has heightened their interest in I/S," said Isenberg.

How much is riding on e-commerce?

Newport Group Inc., Barnstable, Mass., recently completed a study centered on top-tier banking, financial services and insurance industries. All of the participants involved with the study had mission-critical, transaction-based Web applications deployed. They also reported assertive plans toward assimilating electronic commerce into their I/S structures.

On average, survey respondents control I/S budgets that range from $7 million to $50.5 million annually, with an average of 10% of the firm's total budget currently allocated to electronic commerce applications. Furthermore, respondents estimated that the percentage of their firm's total I/S budget devoted to the development, implementation and maintenance of transaction-based Web applications would increase slightly to 13% in 1999, to 23% by the year 2000, and to nearly 30% of the firm's total I/S budget by 2005.

The survey also found that the overall QA process remained constant whether firms were testing client/server applications or transaction-based Web applications. However, a few participants reported that developing Web applications helped to formalize their testing environments. Moving beyond the process, participants were also asked to rate, in order of importance, specific testing practices for client/ server applications and transaction-based Web applications. Load testing gains new attention, in fact, on the Web.

While the importance of specific practices for testing Web applications has changed, the message is clear: There is a tremendous benefit in applying a QA process -- and that knowledge is easily transferable from client/server to Web. Companies simply need to reprioritize the order in which they conduct specific testing practices. For example, client/server applications have traditionally conducted load testing toward the end of the development life cycle. Web applications, on the other hand, require that testing much earlier in the cycle.

Tool choice

For each of the specific testing practices (debugging, load, functional, regression, performance and stress), many mainline testing tool vendors encompass all (or almost all) of these practices into suites of products, many of which now include solutions for Y2K testing. The majority of mainline testing tool vendors began offering Y2K solutions in response to market demands. Currently, these same vendors are leapfrogging one another with maturing solutions for testing electronic commerce applications.

One such vendor covering the enterprise with its automated testing solutions is Mercury Interactive Corp., Sunnyvale, Calif. Ahead of other vendors in terms of "pure testing revenue," Mercury offers solutions for testing client/server, Y2K, electronic commerce, packaged applications and, most recently, applications affected by the European Monetary Conversion. Paramount to its 41% revenue growth to $76.7 million 1997, Mercury decided to partner with enterprise resource planning (ERP) vendors such as SAP America Inc., Wayne, Pa., PeopleSoft Inc., Pleasanton, Calif., and Baan USA Inc., Menlo Park, Calif., in order to offer targeted testing solutions that work to speed the implementation of these packages. With the growth of the ERP market and corporate management's buy-in for this technology, Mercury has gained a strong venue through which to distribute its products. End users have also gained faster ERP implementations, as well as a method to continually test configuration complexities .

On the e-commerce testing front, Mercury recently introduced Astra QuickTest (Q2/'98). Centered on functional testing for e-business applications, Astra QuickTest is an icon-based application, meaning it has replaced traditional scripts with icons to simplify the testing process.

"Utilizing Astra QuickTest cut the time it took to develop tests for our business-to-business Web applications by about two-thirds," said Steve Purpura, senior manager of software development and quality assurance for Comcast Cellular Communications Inc., Philadelphia. "The major advantage is that creating scripts has advanced to a point-and-click activity and requires no scripting."

Purpura chose Mercury's tools over other leading vendors because he felt "Mercury had the best testing solution for the Windows NT platform." As an early adopter of Mercury's LoadRunner and WinRunner tools, Purpura said the value-add for implementing Astra QuickTest is the tool's ability to tie both load and regression tests together into one larger test to run against the Web application.

Utilizing a data-driven approach, Astra QuickTest captures transactions as they are accessed through a browser of choice (including differing versions of the same browser) and then generates multiple test cases (represented by icons) to reflect the real-life actions of multiple users. The tool maps transactions and verifies application functions, such as the loading of HTML tables populated by either static or dynamic data, link functionality, images and Java applets. Test results are then reported directly to the browser in HTML format.

Additionally, Purpura said that Astra QuickTest was the only tool he found that could handle realistic simulation of users when numerous security options were exercised. For example, Comcast employs multiple levels of data encryption via digital certificates and Secure Socket Layers (SSLs), which are all supported by Astra QuickTest. With updates and changes to Comcast's Web applications daily, Astra QuickTest allows Purpura to "build and deploy proactive Web applications with confidence in their functionality." Purpura also uses Astra QuickTest to record advanced application problems reported by users via Comcast's help desk. The problem can then be saved in a file and played back for the development team in order to provide a fix.

With the help of technical services firm Noblestar Systems, Falls Church, Va., SLM Holding Corp., Reston, Va. -- or Sallie Mae, as it is commonly referred to -- implemented SQA Suite from Rational Software Corp., Cupertino, Calif., in order to test its 400+ page Internet site, www.salliemae.com. Recently privatized, Sallie Mae is a leading provider of financial services and the largest provider of student loans, serving more than 900 financial and educational institutions, as well as state agencies. The demand for information through the Sallie Mae Web site is high because it provides instant access to loan information to all of the parties involved in the student loan process. In addition to general information, the site can provide responses on individual loan queries, such as payment due dates or payment receipt postings. The site also contains a calculator function to calculate terms on differing loan types to help students find the loan that best meets their needs.

Because of the rapid changes on the Web site, as well as the continuous addition of information from various departments within Sallie Mae, "an automated tool was necessary in allowing [us] to achieve greater test coverage," said Carol Withers, director of application development. Withers admits that "it was a challenge to win support of the automated tool from the QA team, but once they were adequately trained, they found the automation invaluable."

Withers estimates the current percentage split between automated and manual testing methods for the Web site to be 60/40, respectively. As the Web site grows into more of a dynamic transactional-based site, Withers estimates the amount of automation in Sallie Mae's testing practices to increase further.

Atlanta-based Harbinger Corp., a provider of e-commerce software, services and solutions for supply-chain management, was faced with limited internal resources for its QA practices. The company looked to IMI Systems, Melville, N.Y., a wholly owned subsidiary of Olsten Corp., to fashion a solution for its QA initiatives. "The draw to IMI was their ability to deliver product and methodology," said Karl Lewis, manager of software process and methodology for Harbinger. IMI has designed a proprietary methodology, called Radstar, that is essentially a table-based front-end tool that facilitates the management of test cases and alleviates the need for repetitive scripting. The underlying engine for conducting the regression and functional testing is Automated Test Facility (ATF) from Softbridge Inc., Cambridge, Mass., a wholly owned subsidiary of Teradyne, Boston.

Business units of Teradyne, Softbridge and RSW Software, Watertown, Mass., have also combined development efforts to deliver e-Test Suite, a group of products for testing e-business applications. A notable component to e-Test Suite is Web Monitor. Web Monitor performs non-invasive, post-deployment Web monitoring functions on a 24x7 basis to report problems and provide feedback to organizations on the health of their Web applications.

Customers utilizing Web Monitor should find that the application provides immediate results as to the quality status of their deployed Web applications and provides them with a means to respond to problems more rapidly. There are user-definable options for error handling and message generation. If an error is detected, Web Monitor can automatically send E-mail, for example, to a pager or log the error for tracking by a system management console such as Tivoli or CA-Unicenter.

One end user at a major financial institution reported that utilizing Web Monitor on all of his organization's mission-critical financial Web applications over the past five months "has alerted him to problems early enough to avoid extended outages of his Web application on several occasions." The remaining two components of e-Test Suite are Web Analyst (functional and regression testing) and e-Load (load, stress and performance testing).

The e-Test Suite of Web testing tools, which originated from RSW, is also sold by Cyrano, located in Paris and Newburyport, Mass. Cyrano includes what it dubs Cyrano WebTester with a larger suite of QA tools aimed at addressing standards definition, process management and client/ server testing practices. One other major vendor offering solutions in the automated testing tool space is Compuware Corp., Farmington Hills, Mich., with its QA- Center product.

For most organizations, offering e-business applications simply means adding another component to an overall I/S structure. While the importance of testing has been a staple for all applications -- whether they are mainframe-, client/ server- or Web-based -- the responsibility of adequately testing e-business solutions has become more pronounced because of the "public" nature of these applications. Harbinger Corp.'s Lewis makes an important point with regard to automating the testing process: "Executive management must understand that even though the short-term investment looks expensive, the long-term value will come to outweigh the up-front cost." In simple terms, the better the application, the more money saved in maintenance and support.

Newport Group expects to see the quality awareness and enhancement trend continue to evolve, particularly as it relates to electronic commerce. Web applications are complex, transaction volumes are growing exponentially, and the speed at which these applications must be brought to market influences the need for both testing practices and tools. More importantly, the expanded user base for transaction-based Web applications has growing expectations for Web-offered services. If organizations fall short in meeting these growing user expectations, they risk losing e-business services customers; after all, the competition is only a few mouse clicks away.