News

Quest for software quality: former test manager says you can learn from the past

• By Rich Seeley
• January 5, 2005

When will this application be done? When can we test it? When will the bugs be fixed?

These are questions that have haunted software developers since FORTRAN was in flower. And for almost that long, Michael Mah has been seeking answers.

Back during the Cold War, Mah was working on the guidance system for Trident nuclear submarines, a project for which failure was not an option and quality control was not an afterthought.

"I was the test manager and I was responsible for that very difficult and turbulent last third of the project where we had to debug the software before it was released to the Navy to go out on field trials on a submarine," he recalls. "We had to get all the bugs, find'em and fix'em. We didn't know how many bugs there would be in this very complex software. So we needed to estimate where this software development lifecycle was going with a reliable and accurate prediction on the date that it would be ready and how many bugs we would find and fix so we could tool up with the right staff to do all the testing."

He was still looking for answers when he met Larry Putnam, who is now chief scientist at Quantitative Software Management (QSM), a company specializing in software measurement and estimating tools and techniques. Putnam had a background that combined statistical analysis with nuclear physics, Mah recalls, and he had been working on the same problem of accurately estimating software projects for the military.

Putnam had been head of IT for the U.S. Army Labs and "got creamed in budget negotiations because he was not able to defend his project estimates," Mah says. "So he started researching how software projects behave by collecting a large amount of project data on finished projects, so he could understand how to estimate new projects."

Mah eventually joined him in the effort and today is managing partner of QSM Associates Inc., where a passion for prediction has resulted in products to help software developers and managers move beyond guess-timates.

"Taking an engineers' approach to this problem," he explains, "we wanted to be able to manage by the numbers."

What they envisioned was crunching numbers for data from past software development projects to make predictions on future projects. Building a worldwide database of development information, which currently includes about 7,000 software projects, the QSM engineers led by Putnam created mathematical predictive formulas that can be used to "estimate projects, measure their productivity," and provide what Mah calls "in-flight controls" so programmers can make adjustments throughout the development lifecycle.

The current result of taking all this data and working the math, is a tool called SLIM, which stands for Software Lifecycle Management, he explains. Organizations use SLIM to build their own metrics repository and make relevant industry comparisons, a critical part of project estimation and management.

This approach has attracted corporate customers such as Motorola, Allstate, and JPMorganChase. Enterprise developers and managers use SLIM to determine reliable benchmarks of performance, and see how their quality efforts match up with other companies in their industry, Mah says. Organizations use SLIM in three critical stages, he explains: estimating costs and schedules for new projects; benchmarking completed projects against others; and measuring and controlling projects "in-flight" to prevent overruns.

KeyCorp, one of the nation’s largest bank-based financial services companies, has been using SLIM for more than a year to get more accurate project estimates, says John Stewart, senior VP, enterprise project management for Key Technology Services. He credits the QSM technology with reducing the time development team members need to create an estimate by 20-30 percent with that productivity gain expected to increase.

Matt Light, research director, Gartner Inc., says productivity gains from tools such as SLIM are important because: "Often, IT organizations find that achieving accurate estimates of application delivery projects is either impossible or too difficult and time consuming to undertake. Better defining projects, learning estimation techniques and utilizing tools can dramatically reduce that time and effort while improving project delivery on time and within budget."