In-Depth

Debugging Quality Control

• By Stephen Swoyer
• March 1, 2006

The Big Idea

QUANTIFIABLE CODE

• As regulators clamp down, business users want a hands-on approach to software QA. Some in the industry are pushing for quality-level agreements.
• Developers say traditional metrics don’t always work. Many remain in the dark about their own companies’ internal QA processes. A defined QA process, however, may make their jobs easier.
• A growing trend among Global 2000 companies is to hire quality-control managers and develop QA programs outside their IT depts.

Bugs and other defects are inevitable byproducts of software development. No one disputes this. Beyond that, the consensus usually breaks down.

The disconnect between developers and business users is almost a tradition. Business customers often give short shrift to the concerns of developers. They want software that can handle their business needs, even if those requirements evolve or change. That's why last-minute change requests and eleventh-hour feature or functionality additions are a fact of life for enterprise codejockeys. These down-to-the-wire disruptions may account for defects or bugs that ultimately get fingered by quality assurance testers.

A productive working relationship with line-of-business customers may get even harder to achieve. Sarbanes-Oxley and the current regulatory climate require higher levels of accountability. Not surprisingly, business users want a more hands-on approach to software QA.

As a result, a growing number of codejockeys are grappling with software QA initiatives. They're being asked to take on new responsibilities for testing and other QA-related activities. Why is this a problem?

Because when it comes to QA, many developers are in the dark. Many programmers can't tell you how their companies assess the quality of the software they produce--and in some cases, they don't care to know. Number of X and Y Theodore Woo, a production support engineer with a global IT services firm, is working for a telco that has outsourced most of its IT operations. Woo's firm won its outsourcing pitch by promising to reduce the telco's IT costs and improve the overall quality of its in-house software dev, in part by bringing its developers up to CMMI Level 3 speed.

Neither of these objectives has been met yet, Woo says. And when they are met, exactly how improvements will be measured is unclear. Proposals for software-quality standards, much less quality-level agreements, have yet to become prevalent in Woo's org.

"There have been discussions about this, but whenever it comes up, it's dropped because no one can agree on any workable metrics," he says. Developers tend to scoff at traditional quality metrics, such as X number of bugs or defects per Y lines of code, defects per week or tests per day. Metrics of this kind are useless in many cases, veteran codejockeys say. And in other respects, they argue, such metrics are completely inapposite.

"We have no metric that I'm aware of that measures errors per line of code," Woo says, "but the problem with our type of object-oriented programming is that counting lines of code doesn't have much validity--unless you export everything to one huge text file and basically count it there."

Other metrics include cyclomatic complexity, a widely used static software measurement that in some cases can be used to analyze a software dev project. The more complex a piece of software is, the more forgiving an org might be of defects in the code. Or so the theory goes.

One metric Woo's org uses is functionpoint analysis. It, too, purports to measure the size and complexity of a software dev project. Function points are correlated with app features or requirements. Software QA testers can point to Xbugs per Yfunction points, with Yfunction points preferably being a multiple of Xbugs.

"We get a request to assign a number to what we produce, but none of us know what this is supposed to represent," Woo says. "A group in Mexico is actually performing the analysis. They're the ones who run the numbers on this stuff. But there hasn't been any feedback between us."

QA on the brain
Most orgs perform some kind of internal QA process. However, it's not clear how well rank-and-file developers such as Woo understand this process.

"The idea that a level of quality can be identified easily is kind of ludicrous," says Ken Auer, a principal with custom software dev house Role Model Software.

Auer faces QAexpectations in his own work. In most cases, it's just an inconvenience. "I've been asked to make sure that we ship with stuff like 0 category-1 bugs, less than 5 category-2 bugs, less than 30 category 3-bugs and so on, with some vague description of what each of the categories are," he explains. "This kind of thing is fine, as long as there is an acceptance test period where bugs are identified and adequate time is taken to remove them...or lessen their severity."

Business execs have software quality on the brain. Want proof? Consider the popularity of standardization efforts such as CMMI and ISO 9001. To a large extent, execs embrace these initiatives because they promise to bring a degree of order, regularity or repeatability to a software dev process that to the unpracticed eye can appear anarchic and unpredictable.

In the age of Sarbanes-Oxley, process and repeatability have become more important than ever to corporate decision- makers.

New watchdogs
Software QA is traditionally aligned more closely with IT than with the bread-and-butter business. This isn't an ideal relationship, however, especially in view of IT's role as QA watchdog. To avoid conflicts of interest, some proponents say, QA needs to be spun off as an independent entity with its own chain of command.

Over the last few years, a number of companies have done just that, introducing discrete QA programs and creating positions for quality managers. This trend is becoming the norm among the Global 2000 set.

Mark Eshelby, quality solutions product manager for testing-tools specialist Compuware, says: "We've typically had resources in an organization that are responsible for running tests or building test scripts. And the quality manager is the person responsible for orchestrating that team's work efforts. It's actually becoming very popular."

Compuware markets a line of QAtesting tools such as QACenter and CARS 5.1. IBM's Rational Software Group, Mercury Interactive and many other vendors also market QA complements or QA testing tools.

QA testing tools aren't the problem. It's the absence of QA tools that address the concerns of codejockeys and line-of-business customers that's difficult.

It's possible to deliver a fully functioning app--one with no show-stopping defects--that fulfills all major requirements yet fails to address the needs of the line-of-business customer. "I once wrote a program that did exactly what the document said, and that turned out to be a bad thing," recalls one software engineer. "But I had a document that said, 'Looky here, this is what you wanted, and this is what you got.'"

Good enough to ship
Compuware's Eshelby acknowledges that some QA tools and metrics were not conceived with developers or line-ofbusiness customers in mind. This doesn't have to be the case, he argues. Developers, especially, play an important role in QA. Codejockeys need to understand that the deck isn't stacked against them; an effective software quality QA process can make their jobs easier.

"A lot of [QA] comes down to having a defined process that all of the parties in the organization can work together on," Eshelby says. "It could be as simple as the idea that the entrance and exit criteria are agreed upon by all of the parties in the organization.

"One way we work with a customer on a CARS engagement," he continues, "is to try to involve the end users so they understand the requirements that they've been asking for and what they might mean in terms of risk and time."

He points out that when users find out how long it takes to build and test a feature, you often find out it's not really that important to them.

Role Model's Auer believes this kind of QAvetting, predicated on the early identification of app requirements, is misguided. "Let's face it," he says, "requirements are never even close to complete or accurate. If that's the case, what kind of quality are you measuring?"

Auer advocates a different approach. "The best you can do is to go through some sort of acceptance testing, and decide whether you're happy enough with the quality during acceptance testing that...you want to ship it," he says. "Typically the amount of acceptance testing will be proportionate to the cost of a bug slipping through. If a bug is life-threatening, that [cost] will be a whole lot."

Jeffrey Fredrick, a principal with software QA specialist Agitar Software, thinks the nature of QA testing is going to change, especially as companies continue to flesh out their SOA infrastructures and apps become more virtualized.

Fredrick believes the extreme difficulty of downstream testing will drive more companies to adopt developer testing. "This is the only trend I see that offers some relief to testing departments," he says, "by pushing the testing upstream, the quality of the code as it enters testing is much higher, and this in turn will allow [companies] to work more efficiently. At the same time, testing done by developers is something that can be measured, quantified and communicated."

Age of ignorance
QA testing-tools vendors and other QA proponents tend to paint rosy pictures of an app future-scape in which enterprise software dev is vastly improved. But for many codejockeys, things aren't getting better. Consider production support engineer Woo, whose employer laid off a significant percentage of its developer workforce.

As a result, the QA process at his company has changed drastically over the last few years. "We used to have weekly reviews, meetings where we'd actually put code on an overhead and discuss things [the programmer] might have done instead," he says. "I can't remember the last time we did a code review. It's apparently assumed that the developers just know what they're doing. I have no idea whether they're following the documentation protocols."

As for his employer's function-point analysis initiative, Woo pleads ignorance. "As far as I know, I'm not supposed to have that much of a grasp of it anyway."

PHOTO BY BRITT ERLANSON