This is not a test
- By Sandra Taylor
- July 12, 2001
Client/server systems were hard to debug, but there were comforts associated with the technology. The software
had matured to the point of relative stability, plus the systems were generally restricted to a predefined and
internal user population that could be trained in system policies and procedures. Similar comforts are not associated
with Web technology. Information systems (I/S) groups must now deal with a host of new issues, including browser-based
clients. In addition, Java is a young language based on a new programming paradigm comprised of components and
objects. And there are new middleware infrastructures such as COM/COM+, DCOM and Corba. The scene is made more
complex by the fact the user population can range from zero to thousands, and even to millions, at any time. These
issues are set against a backdrop of increasingly complex back-end systems.
Are the testing/debugging tools keeping pace? The answer is twofold. The world of the Web consists of systems
which run the gamut of complexity -- from the basic Web server with HTML pages, to the high-end system with a distributed
application spanning application servers, database servers and, with increasing frequency, a mainframe. The existing
and emerging tools available to test these environments are as different as the environments themselves, both in
form and function. The organizations typically responsible for administering the tests are similarly distinct.
Therein lies the challenge of bringing together the right tools with the right individuals from various groups
in an I/S organization.
Six point releases a year and counting
of "Web time" quickly went from amusing motto, to insider joke,
to cruel mandate. In so-called Web time -- also known as Internet time
or Web years -- time is accelerated until development efforts are measured
in weeks, not months. This is especially true at the highest of Internet
high flyers, Netscape Communications Corp. At Netscape, truly, Web-time
forces are helping to break down the traditional wall between developers
"Internet time equals insane development schedules," said Michael Lopp, Java platform
manager at the Mountain View, Calif.-based Web browser market leader.
"When Navigator 1.0 came out, we created a QA [Quality Assurance] lab overnight," Lopp
said. It was simple in the beginning. "We had people just surfing the Web, clicking on links, saying if [the
presentation] looked all right to them." But the pressure to decide if a 'build' was good, on 16 platforms,
was mounting. "Now we have a point release every two months. Each needs a full level of testing on the multiple
platforms," said Lopp.
Automation of testing was imperative. "Builds were arriving every day. And we spent half
the day tracking changes in the builds," said Lopp, who turned to Segue Software for some test tools solutions.
Lopp (who's business card includes the whimsical title, "bit sniffer") said he found
Segue's 4Test language sophisticated enough to allow developers to "get creative" in the test sphere.
Moreover, "Segue has the only credible cross-platform story," he maintained.
Today, Netscape is using Intranet technology ["Eating our own dogfood," said Lopp]
to support its test efforts. Bug tracking is Web-based. An Oracle back end is used to store test cases. Lab machines
each run a distributed version of SilkTest or QAPartner from Segue. The QA department can execute unattended automation
runs, and results are posted to all team members via E-mail. "Netscape virtually eliminated manual acceptance
testing using SilkTest and Web-based distributed testing," said Lopp.
Such successes in Web browser testing are particularly welcome these days -- burdened Web servers
and gateways are quickly becoming the next frontier for many of the bit sniffers of the world.
-- Jack Vaughan
The test classes considered below only touch the surface of the available tool inventories. For example, native
testing facilities of various operating systems are not discussed.
Other than Web site and link testing addressed below, QA testing software is the newest of the distributed systems
testing technologies. Typically developed and managed by the QA organization, these tool suites offer the most
deployment trials for a new application. No other class of software has the facility to exercise the entire application
and the underlying infrastructures (middleware, operating systems, networks, databases, etc.) in a real-world environment.
These tool suites, however, are not inexpensive. License costs and, perhaps more importantly, costs to train and
secure the staff needed to run an automated QA effort can significantly increase expenses. Even though test tool
vendors have made major strides in ease-of-use, this is still a case of developing software to test software. And
that is non-trivial.
The investment may be hard to justify for I/S organizations developing applications that are relatively simple
or limited in scope. For a growing number of companies, however, the Net and E-commerce have raised the stakes.
Net-enabled business-to-consumer and business-to-business operations are fast becoming a competitive essential.
A Net-enabled L.L. Bean catalog without a competitive offering from Lands' End? Not likely! Or an online brokerage
system from Charles Schwab while Fidelity sits back and watches? Not on your life! In the world of E-commerce,
the stakes are high, the applications and underlying systems are complex and the mission is critical.
How prevalent is E-commerce? Consider the profiles of the prospective buyers coming through the doors of automated
testing companies. Said Brian Leseur, executive vice president, research and development, Segue Software Inc.,
Newton Centre, Mass., "There's been a noticeable change in the general tenor of the questions we're asked.
In the beginning, we had to spend a lot of time explaining what automated testing was and why it was important.
Now we see companies, driven by intense time-to-market pressures, building very serious applications on very young
technology. These companies are no longer asking why they should test, but how to do it intelligently," commented
Leseur. In support, Jayaram Bhat, vice president of marketing at Mercury Interactive Corp., Sunnyvale, Calif.,
also sees the change. "Companies are willing to invest in QA testing for their E-commerce applications because,
very simply, the cost of failure is so high. With E-commerce, that cost is quantifiable, and in some cases, that
cost is astronomical," said Bhat.
The current mission of automated testing tools is to uncover problems, but probing to find the specific cause
of the problem often lies in the domain of the next two technologies: network traffic testing/ analysis tools;
and program debugging aids.
Network traffic analysis
Response time may well be the Achilles heel of the Net. Time spent staring at messages such as "Web site
found," "Connecting to ... " or "Waiting for a reply" does nothing to ease a user's impatience.
And remember the early HTML implementations? More than a few page authors could not understand why screens were
taking so long to complete. They had no idea that TCP/IP was disconnecting and reestablishing connections between
transmissions. The connections can now be kept open but the example is noteworthy. The programming paradigm is
changing as the era of distributed objects that reside on Net-enabled clients and back-end servers begins. Like
the early HTML page developers, more network surprises await today's developers. In a complex distributed system,
the culprits can lurk in a myriad of places.
System management tools typically provide insight regarding network traffic issues. Problems that lie deeper
within the system and require more in-depth probing, however, must employ a class of tool (in the form of hardware,
software or a combination of the two) known as packet sniffers. Initially, these products offered a fairly rudimentary
capability that produced volumes of cryptic data to represent network traffic. Today's products offer dramatically
improved deciphering and information presentation facilities. Some packages have progressed to the point where
they can be programmed to respond to different conditions, such as a predetermined traffic threshold or a given
number of detected packet failures. The responses include alerts that are automatically sent to network administrators
and describe the offending condition. Some products even watch for packet sequences representative of a hack attack.
Still, other packages operate at a higher level of the network stack and have the ability to capture database-related
protocols such as Oracle's TNS.
Classic debugging tools generally provide the oldest type of testing facility for developers. With the ability
to set breakpoints, step through a code sequence or set/display data, these tools continue to be the mainstay in
the developer's testing tool kit, regardless of whether the tool suite is a 3GL or one of the newer 4GLs. Add the
increasing sophistication of tool suites that scan code looking for usage errors not typically detected by compilers,
and the developer has a formidable arsenal of debugging tools.
Automated QA tests
For those not familiar with automated testing, the
functionality typically falls into two general categories: regression and load testing. Many vendors apply different
names and offer variations on these themes.
Driven from the client perspective, the automated testing software "feeds" defined
user inputs into the application and verifies the application's output matches the anticipated results.
The "defined user inputs" and the "anticipated results" are derived most
commonly through the use of the tool's scripting language. The base theory is that all subsystems involved in the
process are functioning accurately and consistently if the application is generating the correct output.
Most automated testing software
drives the application from the client perspective. With load or stress testing, however, the emphasis is on ensuring
acceptable response times to an increasing number of clients and verifying, to the extent possible, there are no
interaction or timing-related problems.
Perhaps more interesting than the debugging practice itself is what is being debugged. While Java has certainly
spiked the interest of the development community, it must be noted that research from SPG Analyst Services, Natick,
Mass., shows the percentages for Visual 3GLs and 4GLs are also increasing. In fact, respondents in mid- and large-size
companies predict the use of these languages in their companies will more than double over the next two years.
Additionally, the point spread between Java IDEs and Visual 3GLs or 4GLs disappears when respondents were asked
to rank the two classes by perceived importance. The numbers show the two environments were, in fact, dead even
at a 7.65 ranking out of a possible 10.
SPG Analyst Services believes that Java is filling the niche for a general purpose, portable language. Although
it is not commonly acknowledged, the language is fully capable of building a variety of application types and is
not restricted to Net-enabled systems. Unlike many 4GLs and scripting languages, Java is a 3GL. As such, Java is
fully capable of directly accessing operating system primitives (at the expense of portability); interacting with
hardware; or targeting non-traditional data sources such as object-oriented (OO) databases or realtime data feeds.
Java developers have also eliminated some problematic features associated with C++, such as GOTOs, automatic data
type coercions, preprocessing, operator overloading, pointers and others. Lastly, Java's support for automatic
memory management and garbage collection reduces the all-too-common problem of "memory leaks."
For many, Java is a better OO language than C++. Like C++, however, the difficulty of programming in Java can
limit its usability for whole classes of developers. While Java is a simpler programming language than C++, it
is still a 3GL and developers who know and love RAD 4GLs will soon see the difference. Happily, the Visual Java
tools have abstracted much of the programming, using technology such as wizards and templates, and, like 4GL RAD
tools, they ship with a variety of data-aware objects. This still may not be enough; at some point developers will
need to drop down to Java.
JavaBeans provides one solution to this issue when used as a bridge between the minority of technically savvy
Java developers and the business-savvy domain developers. The technically sophisticated developers will construct
JavaBeans. These Beans will be subsequently linked using a 4GL, script-based tool or even a highly abstracted visual
Java tool to build complete applications.
Even in basic Web sites, such as a home page with links to other local pages, things can go awry. Research on
the packet sniffer vendors turned up a site where some of the pages were perfectly readable and others were totally
undecipherable. The pages in error appeared to be text-based HTML documents. Imagine what a few home-grown applets
or a few inter-company links would have generated.
Web sites form the basis for the critical first impression since they are often the first point of contact between
a company and prospective buyers. Make it interesting, easy and fast, and the notoriously less-than-patient electronic
visitor will stay. Make it difficult to navigate, slow or error-prone, and with the click of a mouse the customer
is off to search the competitor's site. Interestingly, these situations represent lost opportunity costs and are
virtually impossible to measure. In the vast majority of cases, a company will never know if prospects leave because
the products are inappropriate or because the site itself is poorly designed with respect to images or content.
The industry can do little to help with a company's products or pricing, but it is helping with site maintenance.
Using tool suites that visually map the Web site and validate links, a Webmaster can ensure paths through the site
are relatively simple and remain intact. On the proactive side, a Webmaster can also trace visitors' paths through
the Web site, tracking metrics such as where they went and how long they stayed on a page. Some products, such
as Segue's Silk, will additionally capture the access time required to walk the page hierarchy. Such timing statistics
have generated more than a few surprises when the development effort was deployed.
Many vendors and industry pundits believe the initial and simplistic use of the Web as a static information
warehouse has become antiquated, and they view such management tools as insignificant.
From their logic, it follows that most companies will progress beyond simple text and animation in their Web
applications to Web-enabled, complex transactional systems. According to this mind set, HTML documents and HTTP
would soon fade into oblivion.
Not likely. Have fat clients and ODBC completely given way to multitiered, partitioned applications and high-performance,
proprietary database drivers? Will they ever? How many DOS users are still out there? For most of us, the only
three things we can count on in life are death, taxes and DOS. Yet no one would seriously argue that two-tier client/server
is effective for large, complex distributed systems. Neither would the debate that ODBC offers better performance
than native database drivers be valid. DOS does not provide better user interaction than Windows, yet it lives
on. HTML and HTTP will also survive the tests of time.
According to Sally Cusack, senior analyst, SPG Analyst Services, "for typical business systems, there will
be a move to more sophisticated functionality. But even in these systems, HTML over HTTP will be used to invoke
that first Java applet or ORB. More importantly, our research shows that browsing for content will not disappear
with the advent of Net-based transactional systems. To the contrary, the amount of static content on the Web will
continue to increase. For this type of information brokering, the ubiquity and simplicity of HTTP and HTML are
all that is required," argued Cusack.
In spite of productivity advances, the industry has yet to discover how to generate error-free software. Software
vendors and I/S organizations face this predicament; it appears to go with the territory. Add today's accelerated
rate of development spurred by time-to-market considerations, and product release decisions become even more complicated.
There is not a single solution to the testing issue. Many of today's systems are just too complex for one person,
or even a development team, to understand the ramifications and interactions of all the hardware and software subsystems.
These subsystems are specialized in functionality, yet for the most part they operate in concert. The I/S teams
that develop and care for these subsystems must perform likewise. Each team, be it development, QA or operations,
can provide knowledge and experience that will make the delivered application both usable and reliable. Said Oliver
Stone, director of Internet products at Rational Software Corp., Santa Clara, Calif., "The Web is the only
environment I know of where real success can become a quality nightmare." Preparing applications for that
success will require cross-discipline efforts from the entire I/S team and high performance testing tools are available
to support the endeavor.
Segue Software at AT&T
Representative Automated Testing Vendors
- Mercury Interactive
- Radview Software
- RSW Software
- Segue Software
- Sun Microsystems
Representative Network Analysis Vendors
- Accrue Software
- Cinco Networks
- Computer Associates
- Network General
- Optimal Networks
Representative Software Development
- Sun Microsystems
Even though the name has changed over the years, AT&T built a telephone network that is the most reliable electronic phenomena in the world. Today, like all companies, AT&T is constantly looking for competitive openings. E-commerce was a natural for this company, but what evolved was E-commerce with a definite AT&T twist.
AT&T's new SecureBuy electronic storefront (www.securebuy.com) was designed for companies who want to conduct business over the Internet but, for various reasons, do not or can not develop the complex transactional software essential to this environment. Mike Maney, E-commerce spokesman for AT&T, headquartered in New York City, described that "this endeavor was almost a natural for AT&T. We've been bringing consumers and business together via voice lines for a long time. With SecureBuy, we're changing the medium to the Internet. That's the idea, but obviously it's not quite that simple. We supply the electronic storefront where our merchants can display their catalogs, we provide the transaction engine, we take care of authenticating and ensuring the security of credit card purchases. We can even help our merchants design their Web pages. And once an order's been placed, our electronic customers can come back to check the status of their order -- what's been shipped, what's back-ordered, etc.," explained Maney.
This is a high-exposure application by any metric. AT&T's name is on the storefront, so browsing consumers know that AT&T is driving the transaction. Subscribing merchants, like Rubbermaid and Better Homes and Gardens, are also investing in the AT&T reputation for reliability and dependability.
Sharing the responsibility for the quality of the underlying software is David Sherman, technical manager, service testing and market trial integration at AT&T's Internet applications services laboratory. While Sherman's title is a mouthful, his task is easily defined -- make sure the software works correctly. No pressure there!
The bottom line
AT&T is a microcosm of the E-commerce evolution as it symbolizes new business opportunities presented by this unique medium. For companies like AT&T who are willing to step into the ring, the true potential is just beginning to appear. For example, the company already envisions expanding its line of services.
The inherent risk in using what still amounts to an emerging technology is one of the downsides. AT&T's reputation for quality and reliability is on the line. Still, the investment in Sherman's group and the Silk automated testing facility is but one example of the commitment to maintain that reputation.