- By Billie Shea
- June 25, 2001
Since its emergence into its own technology discipline more than a decade ago,
automated testing tool vendors have struggled to make their voice and message
heard. Automated testing is gaining more attention as twin forces -- the need
to grapple with year 2000 (Y2K) dates in legacy code, and the desire to do commerce
on the World Wide Web -- confront application developers.
Consider the complexity involved in the deployment of transaction-based Web
applications. Multiple software variables such as browser types, HTML, XML,
communication protocols, operating systems and back-end databases combine with
various hardware platforms and network structures. These variables connected
to Web application development and deployment all represent points of failure
for continuously available Web applications. Strengthening the case for quality,
Web applications represent the integrity of an organization's business operations
and expose the organization to a large audience of external users (for example,
customers and business partners). It is for these reasons that organizations
seeking to absorb electronic commerce into their existing I/S structures have
an increased re sponsibility to ensure the stability, reliability, transaction
integrity and overall quality of the applications developed for conducting e-business.
On another front, the Y2K crisis has pushed organizations to examine and fine-tune
their quality assurance (QA) practices and to define (and secure) specific budgets
for them. For all of the expense and anxiety the Y2K crisis has caused, it has
done well to inspire operational changes throughout many an I/S department.
First, Y2K is classified as a business problem that requires a technical fix.
It has forced senior executives to take a closer look at their company's technology
underpinnings in order to justify budgets for Y2K compliance. Moreover, Y2K
has forced I/S to inventory software, detail a solid plan of action and employ
Still, convincing Corporate America of the value in adopting automated tools
and formal QA practices has been nothing short of an uphill climb. Conceptually,
the quality messages and value to the business are compelling. I/S teams believe
in enhancing the quality of their applications, but have often found that justifying
the price of an automated tool and the cost of formalizing a practice (for example,
employing a QA team to utilize the tool and manage the process) are obstacles.
Furthermore, when applications are pushed for deployment, testing is a good
place to trim when time runs short.
"Quality issues have definitely grown and [end-user] expectations keep
getting higher and higher," said Herb Isenberg, quality assurance architect
for Oacis Healthcare Systems, Greenbrae, Calif. Isenberg has been using automated
tools from AutoTester Inc., Dallas, for the past decade and believes the Y2K
issue has been a catalyst for the heightened focus on quality. "Executive
management is cutting the big checks to resolve the compliance issue and they
want to ensure that the issues are resolved, which has heightened their interest
in I/S," said Isenberg.
How much is riding on e-commerce?
Newport Group Inc., Barnstable, Mass., recently completed a study centered
on top-tier banking, financial services and insurance industries. All of the
participants involved with the study had mission-critical, transaction-based
Web applications deployed. They also reported assertive plans toward assimilating
electronic commerce into their I/S structures.
On average, survey respondents control I/S budgets that range from $7 million
to $50.5 million annually, with an average of 10% of the firm's total budget
currently allocated to electronic commerce applications. Furthermore, respondents
estimated that the percentage of their firm's total I/S budget devoted to the
development, implementation and maintenance of transaction-based Web applications
would increase slightly to 13% in 1999, to 23% by the year 2000, and to nearly
30% of the firm's total I/S budget by 2005.
The survey also found that the overall QA process remained constant whether
firms were testing client/server applications or transaction-based Web applications.
However, a few participants reported that developing Web applications helped
to formalize their testing environments. Moving beyond the process, participants
were also asked to rate, in order of importance, specific testing practices
for client/ server applications and transaction-based Web applications. Load
testing gains new attention, in fact, on the Web.
|Relative importance of
|Table: Source=Newport Group, Inc. 1998
While the importance of specific practices for testing Web applications has
changed, the message is clear: There is a tremendous benefit in applying a QA
process -- and that knowledge is easily transferable from client/server to Web.
Companies simply need to reprioritize the order in which they conduct specific
testing practices. For example, client/server applications have traditionally
conducted load testing toward the end of the development life cycle. Web applications,
on the other hand, require that testing much earlier in the cycle.
For each of the specific testing practices (debugging, load, functional, regression,
performance and stress), many mainline testing tool vendors encompass all (or
almost all) of these practices into suites of products, many of which now include
solutions for Y2K testing. The majority of mainline testing tool vendors began
offering Y2K solutions in response to market demands. Currently, these same
vendors are leapfrogging one another with maturing solutions for testing electronic
Introducing "scenario testing"
A unique approach to automated testing that is now being offered in
response to the need for testing mission-critical e-business applications
is LiveQuality from Segue Software Inc., Newton, Mass. Built on top of
Segue's existing Silk product line of Web testing tools, LiveQuality centers
on scenario testing, which basically blends individual testing practices,
such as load testing and functional testing, into one encompassing system-level
test. LiveQuality consists of three components -- LiveQuality Producer,
LiveQuality Realizer and LiveQuality Delivery -- and seeks to provide
more test coverage on an application, faster. LiveQuality Producer is
the tool (essentially, Silk) that creates test scripts; LiveQuality Realizer
orchestrates the creation and execution of test scenarios, and then monitors
and reports test results; and LiveQuality Delivery renders the consulting/implementation
To deliver the LiveQuality solution, Segue has rallied support from
consulting partners IBM Global Services, Coopers & Lybrand LLP and
Interim Technology. This hand-holding approach to implementing the LiveQuality
testing solution helps organizations ensure that quality assurance groups
properly design and execute test scenarios that mimic real-life conditions.
The investment in LiveQuality is considerable: Pricing starts at $250,000,
with 40% of that investment in consulting alone. According to Segue, LiveQuality
Delivery will provide, on average, one month of consulting. This allows
the consulting team to build and execute scenarios to test the application,
and also provides training on the solution.
Richard Griswold, vice president of quality assurance, is in the early
stages of implementing LiveQuality for Insurance Technology Services of
America Inc. (ITSA). "Segue was the only vendor that met all of our
specific criteria. LiveQuality was Web-based and offered support for SQL
Server, Windows NT and Internet Explorer 4.0," he said. ITSA is now
gearing up for a large rollout of a Web-based insurance application to
run on kiosk machines that will be placed in select SAM'S Club of America
(a division of Wal-Mart) stores.
Consumers will be able to use the application to comparison shop insurance
prices and to purchase various types of insurance from a stable of insurance
companies. "The concept of scenario testing appealed to our organization
because our application will experience very heavy and unpredictable loads,"
said Griswold. "We need a way to test the load as realistically as
possible before the application goes into production."
-- Billie Shea
One such vendor covering the enterprise with its automated testing solutions
is Mercury Interactive Corp., Sunnyvale, Calif. Ahead of other vendors in terms
of "pure testing revenue," Mercury offers solutions for testing client/server,
Y2K, electronic commerce, packaged applications and, most recently, applications
affected by the European Monetary Conversion. Paramount to its 41% revenue growth
to $76.7 million 1997, Mercury decided to partner with enterprise resource planning
(ERP) vendors such as SAP America Inc., Wayne, Pa., PeopleSoft Inc., Pleasanton,
Calif., and Baan USA Inc., Menlo Park, Calif., in order to offer targeted testing
solutions that work to speed the implementation of these packages. With the
growth of the ERP market and corporate management's buy-in for this technology,
Mercury has gained a strong venue through which to distribute its products.
End users have also gained faster ERP implementations, as well as a method to
continually test configuration complexities .
On the e-commerce testing front, Mercury recently introduced Astra QuickTest
(Q2/'98). Centered on functional testing for e-business applications, Astra
QuickTest is an icon-based application, meaning it has replaced traditional
scripts with icons to simplify the testing process.
"Utilizing Astra QuickTest cut the time it took to develop tests for
our business-to-business Web applications by about two-thirds," said Steve
Purpura, senior manager of software development and quality assurance for Comcast
Cellular Communications Inc., Philadelphia. "The major advantage is that
creating scripts has advanced to a point-and-click activity and requires no
Purpura chose Mercury's tools over other leading vendors because he felt "Mercury
had the best testing solution for the Windows NT platform." As an early
adopter of Mercury's LoadRunner and WinRunner tools, Purpura said the value-add
for implementing Astra QuickTest is the tool's ability to tie both load and
regression tests together into one larger test to run against the Web application.
Utilizing a data-driven approach, Astra QuickTest captures transactions as
they are accessed through a browser of choice (including differing versions
of the same browser) and then generates multiple test cases (represented by
icons) to reflect the real-life actions of multiple users. The tool maps transactions
and verifies application functions, such as the loading of HTML tables populated
by either static or dynamic data, link functionality, images and Java applets.
Test results are then reported directly to the browser in HTML format.
Additionally, Purpura said that Astra QuickTest was the only tool he found
that could handle realistic simulation of users when numerous security options
were exercised. For example, Comcast employs multiple levels of data encryption
via digital certificates and Secure Socket Layers (SSLs), which are all supported
by Astra QuickTest. With updates and changes to Comcast's Web applications daily,
Astra QuickTest allows Purpura to "build and deploy proactive Web applications
with confidence in their functionality." Purpura also uses Astra QuickTest
to record advanced application problems reported by users via Comcast's help
desk. The problem can then be saved in a file and played back for the development
team in order to provide a fix.
With the help of technical services firm Noblestar Systems, Falls Church,
Va., SLM Holding Corp., Reston, Va. -- or Sallie Mae, as it is commonly referred
to -- implemented SQA Suite from Rational Software Corp., Cupertino, Calif.,
in order to test its 400+ page Internet site, www.salliemae.com. Recently privatized,
Sallie Mae is a leading provider of financial services and the largest provider
of student loans, serving more than 900 financial and educational institutions,
as well as state agencies. The demand for information through the Sallie Mae
Web site is high because it provides instant access to loan information to all
of the parties involved in the student loan process. In addition to general
information, the site can provide responses on individual loan queries, such
as payment due dates or payment receipt postings. The site also contains a calculator
function to calculate terms on differing loan types to help students find the
loan that best meets their needs.
Because of the rapid changes on the Web site, as well as the continuous addition
of information from various departments within Sallie Mae, "an automated
tool was necessary in allowing [us] to achieve greater test coverage,"
said Carol Withers, director of application development. Withers admits that
"it was a challenge to win support of the automated tool from the QA team,
but once they were adequately trained, they found the automation invaluable."
Withers estimates the current percentage split between automated and manual
testing methods for the Web site to be 60/40, respectively. As the Web site
grows into more of a dynamic transactional-based site, Withers estimates the
amount of automation in Sallie Mae's testing practices to increase further.
Atlanta-based Harbinger Corp., a provider of e-commerce software, services
and solutions for supply-chain management, was faced with limited internal resources
for its QA practices. The company looked to IMI Systems, Melville, N.Y., a wholly
owned subsidiary of Olsten Corp., to fashion a solution for its QA initiatives.
"The draw to IMI was their ability to deliver product and methodology,"
said Karl Lewis, manager of software process and methodology for Harbinger.
IMI has designed a proprietary methodology, called Radstar, that is essentially
a table-based front-end tool that facilitates the management of test cases and
alleviates the need for repetitive scripting. The underlying engine for conducting
the regression and functional testing is Automated Test Facility (ATF) from
Softbridge Inc., Cambridge, Mass., a wholly owned subsidiary of Teradyne, Boston.
Business units of Teradyne, Softbridge and RSW Software, Watertown, Mass.,
have also combined development efforts to deliver e-Test Suite, a group of products
for testing e-business applications. A notable component to e-Test Suite is
Web Monitor. Web Monitor performs non-invasive, post-deployment Web monitoring
functions on a 24x7 basis to report problems and provide feedback to organizations
on the health of their Web applications.
Customers utilizing Web Monitor should find that the application provides
immediate results as to the quality status of their deployed Web applications
and provides them with a means to respond to problems more rapidly. There are
user-definable options for error handling and message generation. If an error
is detected, Web Monitor can automatically send E-mail, for example, to a pager
or log the error for tracking by a system management console such as Tivoli
One end user at a major financial institution reported that utilizing Web
Monitor on all of his organization's mission-critical financial Web applications
over the past five months "has alerted him to problems early enough to
avoid extended outages of his Web application on several occasions." The
remaining two components of e-Test Suite are Web Analyst (functional and regression
testing) and e-Load (load, stress and performance testing).
The e-Test Suite of Web testing tools, which originated from RSW, is also
sold by Cyrano, located in Paris and Newburyport, Mass. Cyrano includes what
it dubs Cyrano WebTester with a larger suite of QA tools aimed at addressing
standards definition, process management and client/ server testing practices.
One other major vendor offering solutions in the automated testing tool space
is Compuware Corp., Farmington Hills, Mich., with its QA- Center product.
For most organizations, offering e-business applications simply means adding
another component to an overall I/S structure. While the importance of testing
has been a staple for all applications -- whether they are mainframe-, client/
server- or Web-based -- the responsibility of adequately testing e-business
solutions has become more pronounced because of the "public" nature
of these applications. Harbinger Corp.'s Lewis makes an important point with
regard to automating the testing process: "Executive management must understand
that even though the short-term investment looks expensive, the long-term value
will come to outweigh the up-front cost." In simple terms, the better the
application, the more money saved in maintenance and support.
Newport Group expects to see the quality awareness and enhancement trend continue
to evolve, particularly as it relates to electronic commerce. Web applications
are complex, transaction volumes are growing exponentially, and the speed at
which these applications must be brought to market influences the need for both
testing practices and tools. More importantly, the expanded user base for transaction-based
Web applications has growing expectations for Web-offered services. If organizations
fall short in meeting these growing user expectations, they risk losing e-business
services customers; after all, the competition is only a few mouse clicks away.