The incredibly unique testing requirements of Web apps
Rapidly expanding Web applications pose new challenges to software developers.
Because they support manufacturing operations in areas such as just-in-time
supply chains, and end-user buying and selling, the accuracy and reliability
of these apps needs to be high. But current software development processes
do not meet all the needs of these new applications, particularly in the
area of quality assurance.
Web-based applications continuously evolve. Network configurations change
and new types of software are added with little overall control. Web pages
are frequently moved, requiring changes to the links of any applications
that use them. Users may upgrade their browsers, and other applications
may be added to the Web to compete for the same resources as existing
applications. Similarly, the hardware and software on Web servers frequently
changes. This creates a need for frequent re-testing of Web applications,
even when the application itself has not changed.
In addition, the Web's competitive environment depends on speed and reliability.
Sites with poorly performance can irritate users, who are only a click
away from the competition. Business transaction errors in Web-based applications
can result in immediate financial loss, as there is no human in the transaction
loop to perform "sanity checks." The speed and volume of Web
business can cause large losses to accrue before errors are even noticed.
Once again, testing is the only defense against damage resulting from
such software errors.
In traditional development environments there is little doubt about the
importance of testing critical applications. There are many potential
approaches to testing, each of which involves a different degree of automation.
Most large projects will use some bespoke tools in the testing process.
For Web applications, however, the cost/benefit balance between automated
and manual testing shifts greatly toward automated testing due to the
amount of repeat testing required. For example, tests need to be performed
on all the platforms the application will run on, under varying levels
of load, and when the application or its environment changes.
The complexity of Web application testing eliminates the possibility
of building bespoke testing tools for a particular project or organization.
However, commercially available tools do not yet address all the concerns
of Web application owners and users.
What needs testing?
Software testing demonstrates that a piece of software is fit for its
intended purpose. This typically includes checking the following:
· The software should deliver the correct results, using enough
sets of inputs to give an adequate level of confidence that it will work
correctly for all sets of inputs it will meet in use ("functional
· The software should perform adequately, handle any foreseeable
workload and deliver its results within an acceptable time while using
an acceptable level of resources ("performance testing").
· The software should conform to criteria from either the organization
that will use the software or from an external organization - such as
a regulatory body, or a customer of the organization using the software.
· The system delivered should be usable in its intended role, considering
human/ergonomic and environmental factors.
· The system should be rugged and reliable, as well as handle the
failure of its own components.
· The data used by the system should be secure.
In addition to the above, Web applications require particular emphasis
on usability, performance and security. Indeed, security is an issue that
has not often been included in the tester's brief in the past.
Link testing and load testing are essential for all Web sites, even when
they are only providing information and not actually transacting any business.
Yet load testing of the type used for traditional architectures is not
adequate for the Web - testers must confirm that they are getting sensible
responses to service requests and not just "server busy" messages.
When an application uses the output from a Web server, it is essential
to check that the application responds to any denial of service in a constructive
way. This can only be achieved by either including a background load element
within the functional testing phase or by specifically simulating all
possible component failures. A limited amount of functional testing needs
to be included in every load test; hence, the distinction between load
testing and functional testing is not clearly defined in the Web application
Technically, Web pages are complex entities. Several mark-up languages
are used to define the content of Web pages. A single page can contain
numerous different types of objects: text, links to other pages, interactive
dialogs, images, audio or video clips, Java applets, ActiveX components
and so on. These exemplify three very different behaviors:
· Fixed objects that do not change while they are displayed.
· Interactive objects that send new output to the user interface
in discrete blocks in response to user input or messages from the Web
· Dynamic objects that continuously change their output without
requiring any external stimulus.
Testing dynamic content requires a new approach. Conventional testing,
whether automated or manual, is based on the comparison of specific outputs.
It is not useful to regard a dynamic output (visual, audio or otherwise)
as a sequence of millions of static outputs even if this is a correct
physical interpretation. A more practical solution is to split the testing
into a two-phase process: Validate that a particular combination of programs,
scripts and data can generate an acceptable output, and test that the
Web application delivers this combination of outputs to the client terminal.
Performance testing is central to Web application testing. However, the
performance of these systems is dependent on the network -merely stress
testing a Web server or an application server will not in and of itself
render useful information. Preliminary work needs to be done to identify
key factors in the performance of the specific application so that they
are simulated during the test. Some testing tools can now simulate the
use of a range of IP addresses in a test session, a useful step toward
making load tests more realistic but one that is still a very short of
a simulation of the network.
The appearance of a Web page is determined by the browser on which it
is displayed, and modified by user options set on the particular browser.
Because the behavior of a browser can be modified using features such
as cookies, content must be tested at a logical level rather than at the
level of comparing screen displays. It may also be useful to test the
operation of an application on a range of browsers.
Over the last two years, testing tool suppliers have made considerable
progress adapting GUI-based tools for use on Web browsers, simulating
load on a variety of servers, and recognizing the interrelationships between
load testing and functional testing. The explosion of e-commerce offers
them the best opportunity they have had. There remain a number of user
needs that are not satisfied by the current offerings. This will keep
vendors busy for some time to come.
Graham Titterington is co-author of Ovum's report E-business security: New directions and successful strategies.