ASQ is more than testing tools
- By Philip E. Courtney
- May 29, 2001
True or false: Automated testing is the only way to ensure the quality
of an application. The answer depends on whom you ask and the depth of
his or her experience. Software developers are told over and over that
testing is the last chance to "get it right" prior to deployment.
Yet a number of tools can be used to improve the quality of business
applications long before the testing process begins and, in many cases,
long before a line of code is written. "You've got to take the definition
of quality and make it much broader," said Carl Zetie, senior industry
analyst at Giga Information Group, Cambridge, Mass. "Some organizations
unfortunately equate quality with software that doesn't crash. Instead,
they should think about 'fitness for purpose' and trace development activities
back to requirements."
Development organizations that understand requirements can effectively
model apps and build test cases that accurately validate the features/functions
that meet their business needs, say observers. Applying automated software
quality (ASQ) disciplines early in the development life cycle can improve
the quality of the application and, over the long run, reduce correction
"Statistically, 60% to 70% of application failures can be attributed
to problems early in the life cycle," said Graham Titterington, senior
analyst at Ovum Inc., a London-based consulting firm. "The most important
aspect of quality application delivery is using the correct process that
helps provide early notification of a problem."
For many programmers, though, "process" produces an effect somewhat
similar to the way "fur" provokes an animal rights terrorist. Furthermore,
faced with time-to-market pressures, "process" is typically envisioned
as a stumbling block to creativity.
"Software developers have an aversion to process. They think about programming
as a craft," said Randy Rice, president of Rice Consulting Services, Oklahoma
City. "A consistent approach is needed because quirky software leads to
quirky testing, and non-standard disciplines cause problems to ripple
outward from the beginning of the project."
Thus, many consultants do away with the term "process" in favor of monikers
like "framework" and "workbench." Or perhaps "methodology," as is the
case at Stockholm, Sweden-based telecommunications giant Ericsson. Developing
hardware and software components from its Network Core Products group,
Ericsson relies on an automated development methodology that flows from
requirements through development and testing, and into production deployment.
"The need for an automated methodology is very important for the delivery
of quality products," said Lars Taxen, a member of the Ericsson team responsible
for implementation and support of new network products. "It is especially
important for us because we have 10,000 design engineers."
It is also important because development requirements in the highly
competitive telecommunications market are never fixed, requiring flexibility
without sacrificing quality, noted Taxen. While fluid requirements can
give most programmers fits, Taxen said the eMatrix toolset from MatrixOne
Inc., Chelmsford, Mass., has let Ericsson implement automated collaborative
and incremental development practices while tracing requirements through
modeling and testing.
"Incremental development is guided by requirements and allows us to
develop and test applications in smaller, more meaningful portions," Taxen
said. "We then expand the methodology to test greater numbers of increments
prior to a product rollout."
In effect, Ericsson expanded the definition of automated software quality
by wrapping the disciplines of requirements traceability, modeling, incremental
development and testing into a single comprehensive methodology.
Defining the requirements
In the best of worlds, the quality engineer would be involved as early
as possible in the application development life cycle - usually during
the requirements definition stage because open communications between
the QA and development teams help build product components that are truly
'testable,' said Giga's Zetie. "How can you test a requirement that reads
'the product must be easy to install'?" he said.
Just as the quality of an application is dependent upon extensive testing,
the quality of testing is dependent on the extent of the requirements.
"Testers must build test cases based on application requirements," said
Mary Walker, knowledge engineer at Logica North America, Lexington, Mass.
"Otherwise, there's a lack of communication on what actually needs to
be validated." Using the RequisitePro toolset from Rational Software Corp.,
Cupertino, Calif., Logica records application requirements from a variety
of sources, including clients, field personnel, technicians and others.
These recorded requirements are reviewed by software architects, developers
and quality assurance staff in an effort to determine resource needs and
- deemed equally important by the company - testing requirements.
"As part of the process, testing requirements are defined and approved
based upon application requirements," said Walker. "No code is written
prior to review and approval by all groups. The added communication helps
everyone build the best quality product possible."
Another company moving automated quality disciplines to earlier in the
development life cycle is ESPS, a developer of custom publishing software
for a number of industries that must meet strict government regulation
demands. "We require a much more structured development environment,"
explained Bob Schatz, executive director of development and operations
at the Ft. Washington, Pa., firm. "It was critical that we captured all
of our requirements and link them with development and testing activities."
The need for requirements traceability became evident for ESPS shortly
after the first release of its product, Core Dossier. According to Schatz,
ESPS must respond to the guidelines and auditing requirements of its customers,
which are directed by government entities such as the Federal Drug Administration
(FDA) or the Environmental Protection Agency (EPA). "We had a long way
to go after the first release of our product," he said. "Instead of trying
to find a way to re-create requirements from memory, we implemented a
tool that helped us reverse engineer our existing functionality and build
a baseline of requirements."
ESPS implemented the DOORS toolset from Quality Systems and Software
Inc., Mt. Arlington, N.J., to reverse engineer existing software into
functional requirements. Additionally, the company divided and defined
requirements based on product lines, first creating a baseline for the
core components contained in each product, and then creating and linking
separate requirements for the various attributes of each individual product.
"While we maintain one set of requirements for the base product, the
requirements differ across the add-on products," Schatz said. "Now they
are all linked in the DOORS database so that we can immediately learn
the impact of changes across different product lines."
According to Ovum's Titterington, the value of requirements management
tools can only be fully leveraged when open communication exists between
the various IT organizations. "It's part of an inspection and review process
where objectives are clearly stated, documented, understood and followed
through," he said.
Titterington also noted that while automated integration with modeling
and test-planning products does improve the software delivery methodology,
manual decision points still exist along the way. For example, while Web
front ends to such tools allow more users to enter requirements, today's
tools cannot determine whether a requirement is useful or contradictory.
"Even with automation, a decision-maker must be involved," he said.
The application modeling discipline pushes the automated software quality
process to earlier in the development life cycle. Typically used following
the creation of requirements, modeling tools can provide support for rapid
application development and help alleviate some ever-present time-to-market
demands. "The ultimate dream is to have a modeling tool produce 100% fully
functional code," said Mike Budd, a senior consultant at Ovum. "A good
tool will provide anywhere from 70% to 80% of code generation."
Modeling can be viewed from several different angles, each with the
goal of reducing app development time and helping to promote reuse of
From the development perspective, modeling helps define the application
flow while generating and managing source code and objects. From the perspective
of a business user, modeling provides an opportunity to define business
functions that require underlying code. An example of the latter rule
is Business Process Template (BPT) from Template Software Inc., Dulles,
Va., which is said to help organizations capitalize on business knowledge
contained within the organization by graphically depicting processes.
"Reusability is the key," said Joe Damassa, vice president of application
development marketing in the Software Solutions Division at IBM, Somers,
N.Y. "Crucial to the delivery of quality applications is ease of use in
application development tools and processes."
Damassa said that ease of use and reusability are the keys to IBM's
oft-changed SanFrancisco e-business framework, which is gaining support
from third-party tool makers like Princeton, N.J.-based Princeton Softech
and its Select Enterprise modeling toolset. Through an automated pattern
transformer, Select Enterprise applies the standard IBM SanFrancisco code-generation
rules to a high-level model and generates a "design-" level view. This
design-level view can be reviewed and modified prior to code generation.
"A fully expanded view of the application prior to code generation or
changes goes a long way to improving the quality of an application," said
Ovum's Budd. "In effect, good use of a modeling tool can help produce
a diagrammatic view of the tasks required by the business. Modeling helps
you define the 'what' of the system, not the 'how.'"
Test planning tools:
At the core of automated quality disciplines reside test-planning tools.
Throughout the years, these tools have continued to evolve to the point
where the name "test planning" has become an almost inadequate description
of the capabilities of most products.
For example, requirements management tools such as Quality Systems and
Software's DOORS and Rational's RequisitePro today have interfaces with
test-planning products such as TestDirector from Mercury Interactive Corp.,
San Jose, Calif., and Rational's SQA Manager. These interfaces are designed
to provide two-way communications so that changes made in one product
are automatically reflected in the other. "Bringing requirements right
into test planning represents a dramatic improvement in quality assurance
capabilities," said Ovum's Titterington.
"Communication between the technologies provides an ideal means for
eliminating any interface constraints," added Giga's Zetie. "It allows
for interface dependence and collaborative conversions as more people
can access the requirements and review the test results."
Outside of the requirements realm, though, integration with other technologies
enables test-planning tools to tout support for functions such as version
management, test-data generation, systems management testing, defect tracking
and an array of reporting capabilities. Mercury Interactive's TestDirector
includes a Web interface that can be used to view all aspects of an app
from requirements to test results. This integration becomes especially
important for those firms deploying packaged apps where requirements and
application models are unavailable. Furthermore, these functions can provide
a centralized view of quality assurance activities.
The "P" word
Whether or not it is called a process, framework or methodology, organizations
serious about quality application deployment have implemented steps to
assure communication between various IT groups, while instilling the disciplines
needed to meet business requirements.
"Regardless of how much of an aversion there is to formal, tool-driven
methods," said Zetie, "there are procedures already in place by most organizations
whether they use E-mail, meetings or MS-Word documents."
At People's Bank in Bridgeport, Conn., the use of a non-tool-driven
process enabled the firm to successfully and automatically test its new
Web-based banking system - complete with fully active streamlining video
- using the e-Test Suite from RSW Software Inc., Waltham, Mass. "We first
identified and reviewed our requirements," explained Steve Guidone, principal,
People's Bank Emerging Technologies department. "Then we built our test
cases based on those requirements and loaded them into the testing tool."
Some of those requirements included validation of typical customer tasks
such as reviewing an account balance, paying bills and funds transfers.
It also included validation of architectural requirements to ensure that
the system could handle anticipated loads. "We began with testing some
of the more non-intrusive functions and rippled the process outward, adding
more as we went along," said Guidone. "The goal was to mimic the application
and business requirements."
In formal situations, disciplines can be driven by tools such as MatrixOne's
eMatrix or IS*Integrity from Chain Link Technologies Inc., Sunnyvale,
Calif., which uses a Web interface to flow requests for change with the
actual change process. This interface automatically links items such as
requirements and defects with approvals for production deployment.
Disciplines can also be embraced with IBM's SanFrancisco which, according
to the firm's Damassa, is delivered with a process and a workbook wrapped
around development tools. "The goal is to take people who are not very
skilled in object and Java technologies and have them learn how to quickly
deliver new applications that contain the quality needed by the business,"
he said. But care must be taken when seeking tools, contends Rice at Rice
Consulting Services, because tools cannot always be made to fit a process.
"Although a consistent approach is needed, it seems that most people have
a product-driven thought process," he said. "The questions always seem
to evolve toward 'What are the features and functions?'."
Ovum's Titterington agrees and adds that IT organizations must avoid
the distinct danger of becoming enamored of technologies without a good
methodology already in place. "Building a process around your tools will
help you end up in tears," said Titterington. "The tools should be selected
based on your requirements as opposed to making your requirements meet
the functions of a tool."
Automating software quality
So, is automated testing the only way to ensure the quality of an application?
No. Instead, automated testing represents a portion of a comprehensive
automated software quality practice that includes requirements definition
and tracking, app modeling, test planning and execution, defect tracking
and change management.
Equally important, although automation can help guide development activities
from concept to finished product, the delivery of business systems will
still require vital, manual intervention for decisions.
"You can't automate design or requirements differences," said Titterington.
"And you can't automate the creation of the business plans, marketing
plans or product launch plans. Automation can assist [you], but it cannot