THINKPIECE: Quality counts, especially in outsourcing
- By Venkates V. Swaminathan
U.S. companies spent approximately
$1.6 billion on custom application development
services from offshore providers in 2003, and that will grow at a 23% annual clip, according
to research firm IDC.
For many companies, outsourcing IT work is
seen as mandatory to stay competitive, and even software product companies are offshoring product development and maintenance work to reduce costs.
But many experts will tell you that outsourcing
software development carries increased risks, most notably, quality. The quality issue, says Melissa Webster, a research director at IDC, is analogous to what the manufacturing industry faced many years ago. At one time, one could
easily tell when something was manufactured
overseas in a low-cost manufacturing
center. Low cost meant low quality.
But all this has since changed, in no small
part because of processes that specify and
measure quality at each stage of the manufacturing process.
For outsourced software development
— especially offshore work — to match
the quality and functionality of in-house
work, it needs stronger quality assurance
processes. Without them, outsourcing
can be penny-wise, pound-foolish.
One solution receiving increased interest
is the quality-level agreement,
which Webster says is a service-level
agreement for application development.
A QLA has two major components:
a set of metrics and measurement
processes to define quality; and
the contractual language that defines
enforcement and penalties.
The concept is simple to understand,
but hard to define, because measuring
quality in software is notoriously difficult.
Tests for end-to-end processes often
depend on configuration and customization
that are not well defined,
and performance and load testing require
setup and infrastructure that cost
significant time and money. Besides,
writing test code is laborious, expensive,
and very hard to maintain.
But advances in technology make the
concept possible. For any CIO or VP of engineering who’s managing a far-flung team,
simple project status reports that say "ontrack"
or "behind schedule" aren’t enough.
What an executive needs is a good sense of
the level of risk: How many scenarios
work? How good is the code? How does it
handle boundary conditions? How well
written is it? How well has it been tested?
The QLA should include these five
1. Require the vendor to deliver a
comprehensive test suite with the application.
As IDC’s Webster says, "If you think of software as an asset, without a suite of tests, you have an uncertain asset." The offshore supplier, she
notes, needs to provide the test suites
that document and verify functionality.
2. Provide for conducting tests and
sharing test results throughout the development
process. Software testing has long followed a waterfall model, in which testing is done near the end of the process. But having it as an ongoing process is crucial to producing
high-quality code at the end, and managing
3. Base test coverage requirements on risk, not just code coverage metrics. Mere code coverage metrics (requiring each line of code to be tested) miss too many areas (boundary conditions,
fallout scenarios, performance). Good testing requires an early and systematic assessment of risk related to code quality as well as business process and performance.
4. Require white-box and black-box
testing. Historically, vendors have been
reluctant to do systematic white-box
testing or to share those results. But the
National Institute for Standards and
Technology says this kind of testing is
crucial for delivering high-quality code,
especially when the development team
is a great distance from the requirements
and analysis team.
5. Require tests that can be independently
conducted and verified. More customers are requiring that tests be conducted not just by the vendor, but also by independent groups. These
tests must cover quality — simplifying
dispute resolution, and ensuring that requirements are thoroughly specified
and tests thoroughly documented.
These technologies, and others like
them, enable CIOs to define reasonably
thorough sets of quality criteria by
which to judge deliverables, and to
measure them accurately enough to
put them in a contract. For customers
looking for ways to address the risks associated
with outsourcing and offshoring,
QLAs could be useful.
Venkates V. Swaminathan is a consultant to
companies that outsource software