Columns

THINKPIECE: Quality counts, especially in outsourcing

• By Venkates V. Swaminathan
• December 1, 2004

U.S. companies spent approximately $1.6 billion on custom application development services from offshore providers in 2003, and that will grow at a 23% annual clip, according to research firm IDC.

For many companies, outsourcing IT work is seen as mandatory to stay competitive, and even software product companies are offshoring product development and maintenance work to reduce costs.

But many experts will tell you that outsourcing software development carries increased risks, most notably, quality. The quality issue, says Melissa Webster, a research director at IDC, is analogous to what the manufacturing industry faced many years ago. At one time, one could easily tell when something was manufactured overseas in a low-cost manufacturing center. Low cost meant low quality.

But all this has since changed, in no small part because of processes that specify and measure quality at each stage of the manufacturing process.

For outsourced software development — especially offshore work — to match the quality and functionality of in-house work, it needs stronger quality assurance processes. Without them, outsourcing can be penny-wise, pound-foolish.

One solution receiving increased interest is the quality-level agreement, which Webster says is a service-level agreement for application development. A QLA has two major components: a set of metrics and measurement processes to define quality; and the contractual language that defines enforcement and penalties.

The concept is simple to understand, but hard to define, because measuring quality in software is notoriously difficult. Tests for end-to-end processes often depend on configuration and customization that are not well defined, and performance and load testing require setup and infrastructure that cost significant time and money. Besides, writing test code is laborious, expensive, and very hard to maintain.

But advances in technology make the concept possible. For any CIO or VP of engineering who’s managing a far-flung team, simple project status reports that say "ontrack" or "behind schedule" aren’t enough.

What an executive needs is a good sense of the level of risk: How many scenarios work? How good is the code? How does it handle boundary conditions? How well written is it? How well has it been tested? The QLA should include these five provisions:

1. Require the vendor to deliver a comprehensive test suite with the application. As IDC’s Webster says, "If you think of software as an asset, without a suite of tests, you have an uncertain asset." The offshore supplier, she notes, needs to provide the test suites that document and verify functionality.

2. Provide for conducting tests and sharing test results throughout the development process. Software testing has long followed a waterfall model, in which testing is done near the end of the process. But having it as an ongoing process is crucial to producing high-quality code at the end, and managing risk throughout.

3. Base test coverage requirements on risk, not just code coverage metrics. Mere code coverage metrics (requiring each line of code to be tested) miss too many areas (boundary conditions, fallout scenarios, performance). Good testing requires an early and systematic assessment of risk related to code quality as well as business process and performance.

4. Require white-box and black-box testing. Historically, vendors have been reluctant to do systematic white-box testing or to share those results. But the National Institute for Standards and Technology says this kind of testing is crucial for delivering high-quality code, especially when the development team is a great distance from the requirements and analysis team.

5. Require tests that can be independently conducted and verified. More customers are requiring that tests be conducted not just by the vendor, but also by independent groups. These tests must cover quality — simplifying dispute resolution, and ensuring that requirements are thoroughly specified and tests thoroughly documented.

These technologies, and others like them, enable CIOs to define reasonably thorough sets of quality criteria by which to judge deliverables, and to measure them accurately enough to put them in a contract. For customers looking for ways to address the risks associated with outsourcing and offshoring, QLAs could be useful.