In-Depth

Tailor-Made for Applications Development

• By Alan Joch
• October 1, 2005

What’s the first thing you do when you have to write a new software program on a tight schedule? For years, the answer has been to just start banging out code.

Now, a growing number of programmers in large commercial software companies, corporate IT departments and the military are dealing with tight deadlines by delaying the time when they write that first line of code. Instead, they spend almost four days analyzing their resources and requirements and perhaps sparring with senior managers to make sure that deadlines aren’t only tight, but that they’re reasonable.

These nonconformists are devotees of Personal Software Process and Team Software Process, two intertwined methodologies that offer formal processes for improving software quality and managing schedules. “This isn’t rocket science or a silver bullet,” says Eileen Fagan, software engineering process manager for Intuit, which began using PSP and TSP early last year. “It’s a collection of all the things you need to do to be highly successful, one project at a time.”

Intuit declines to reveal what it spent for PSP/TSP and what the resulting paybacks were, but it and other organizations believe the extensive data collection associated with the methodologies point to concrete benefits.

The techniques are having an impact on deadline performance and software quality, not only at Intuit, but also at Boeing, IBM, Microsoft, Motorola, Teradyne and the U.S. Navy. In some cases, PSP and TSP are helping teams deliver programs with less than one bug per 1,000 lines of code.

As time goes by
Obviously, software quality has been a driving force for developers, IT managers and senior executives for years. However, with each passing year, as increasingly sophisticated computer systems run everything from the most complex fighter planes to mundane household appliances, the ripple effect of programming problems becomes more acute. Adding to the nuisance—and danger—of quality problems are hackers who exploit security holes in commercial software to steal personal and financial information and feed worldwide identity theft rings.

Like most software bugs, security holes usually are fixed after an intrusion. Now, some commercial developers and software engineering academics say enterprises can no longer adopt the traditional break-fix approach to software development for financial, competitive and security reasons. Instead, software developers, working with support from their managers and senior executives outside the IT department, need to fix code during the development process, which can also help them do a better job of scheduling and cost-estimating large programming projects.

One way to do this is using methodologies that have become common among quality-savvy manufacturing companies. A formal strategy for implementing these practices is the two-pronged PSP/TSP attack being pushed by the Software Engineering Institute, part of Carnegie Mellon University. Watts Humphrey, an SEI fellow and developer of both PSP and TSP, has spelled out these principles in a number of books, including Winning With Software: An Executive Strategy and PSP: A Self-Improvement Process for Software Engineers.

PSP, first developed in the early 1990s, helps individual programmers estimate how long it takes them to write code, document the incidence of bugs and identify defects early in the process to avoid time consuming fixes at the end. The second prong, TSP, arrived at the end of the 1990s and expanded PSP principles to workgroups composed of PSP graduates.

It’s the increased adoption of that second prong among commercial and military software engineers that offers encouraging signs that complex software projects can be better managed. “Until there was TSP to go along with PSP, our goals weren’t fully achievable,” says Jeff Schwalb, National TSP lead for the U.S. military’s Naval Air Systems Command (NAVAIR, which provides support for the maritime combat activities of the Navy and Marines). Creating high-quality software can be a matter of life and death at NAVAIR; programs often bolster aircraft and weapons systems.

It’s like factory work sometimes
PSP and TSP offer an alternative to the decades-old software approach of “banging out code and testing it,” Watts believes. “That approach leads to re-dos. Anyone with any quality background will look at that and say that’s fundamentally wrong,” he says.

With PSP and TSP, he applies quality principles introduced by W. Edwards Deming and Japanese auto makers that focus on quality at the beginning of the job. “The fundamental principle of factory work is you do it right the first time, which makes you cost competitive because you eliminate redundant work,” Humphrey says.

He says software developers haven’t traditionally taken this approach because they’re often pressured by management to meet unrealistic schedules that induce errors. “Senior managers always push completion dates until they get enough push-back to convince them that this is the best they can expect,” Humphrey says. “The problem is software developers haven’t known how to push back” so they find themselves agreeing to deadlines without first determining whether the schedules are realistic. “Basically, all we’ve done with PSP and TSP is to show software developers how to push back,” Humphrey says.

He set out to develop this language in 1989, and over two years, wrote 70 programs. Along the way, “I gathered data on everything, I managed quality and I defined processes—everything I thought should be done,” he recalls. “And I found it worked.” Using statistics on how long it took him to write programs and by documenting defects as he went along, Humphrey could accurately estimate schedules and improve the quality of his programs.

He began teaching PSP courses that mirrored his experiences. Current classes require students to write eight programs using a standard process: Make a plan, develop a program, do a post mortem. “During those steps, students gather data about how much time it takes them to do the work, measure how big the program is and record all of their defects,” he explains.

Although there’s a lot of formal classroom and textbook study associated with PSP instruction, TSP is much less theoretical and more hands-on, says NAVAIR’s Schwalb.

Reconciling needs with desires
TSP consists of a number of project development phases, beginning with a detailed planning session. According to TSP gospel, the development staff first hears a project summary from senior management that spells out the parameters of the new software initiative, including the deadline for completing the code, the business case for launching the effort and any details—often scant—about specific functionality the new software should offer.

Next, the development staff sequesters itself for 3 1/2 days to compare project goals to available resources and to the data collected by PSP methods showing how fast individual staff members can write nearly defect-free code. Then, they reconcile this analysis with senior management’s deadline expectations.

Day 4, when developers present their conclusions to management, is typically when the fireworks fly. Rarely do a development team’s conclusions meet the deadlines set by management, Humphrey says. But the analysis process provides a way for the two sides to hammer out a more workable schedule, preferably with alternatives the development staff identified in advance. For example, the programmers might ask for more resources or cutbacks in the functionality of the proposed software.

“We tell the teams to do their best to do what management wants and prove they really put their hearts into the effort,” Humphrey says. “Out of the hundreds of teams I’ve seen go through this process, I’ve yet to have a team lose an argument, because they have a plan, lots of data and they put a lot of effort into it. If management argues, everyone on the team can come back with a response. When the whole team is there and everyone’s behind the effort, management listens.”

Although the planning phase is a key starting point to TSP, it’s not a one-time effort. Schwalb says large projects typically incorporate multiple planning sessions that coincide with key milestones.

Meeting schedules with fewer bugs
NAVAIR has been using PSP since 1996, after Schwalb heard a presentation by Humphrey a year earlier. NAVAIR added TSP around 2001. The original move to PSP was part of the organization’s overall desire to improve its software development processes, a move that first led it to classical process improvement techniques that included the Capability Maturity Model, a framework developed by the SEI that spells out hundreds of development best practices.

The emphasis of CMM and its cousin CMMI (Integrated) is organization-wide process management, not the individual and team orientation of PSP and TSP.

NAVAIR’s software managers became frustrated with CMM after spending months creating new processes, then finding themselves conducting seemingly endless training session to get everyone onboard. “It was just taking too long, and people were running out of energy,” Schwalb recalls. When he learned about PSP, however, “I made paradigm shift,” he says. “It gave us a starting point and a way to tailor the development process.”

Within about six months of taking SEI-led PSP training courses, many of NAVAIR’s developers began to feel comfortable collecting performance data and using it for project management. Success rates in hitting scheduling targets soon began to improve, Schwalb ways. Although PSP moved NAVAIR a step closer to where Schwalb wanted to go, it wasn’t a complete answer. After going through PSP training, many programmers said they weren’t using what they’d learned because others in the organization weren’t similarly trained. “That was frustrating, but nobody said stop the training, Schwalb recalls. That frustration eased in 2000 when NAVAIR began incorporating the workgroup concepts of TSP that the SEI was formalizing. “We saw real successes when we applied TSPin its full version around 2001,” Schwalb says.

Planning and ongoing timetable assessments have had an impact at NAVAIR. In the past, projects may have come in anywhere from six to nine months behind schedule. The new PSP/TSP techniques have helped the staff hit deadlines within one week of the original schedule, Schwalb reports. What’s more, code quality goes hand in hand with hitting schedules. One reason schedules are more accurate is that NAVAIR is spending less time fixing errors at the end of the development process.

Late fixes are not only time consuming, they’re especially costly because changes often have a ripple affect on other sections of completed code, which then must be revised to accommodate the corrections. “When you go into test phases, you spend lots of unplanned hours or perhaps even weeks on the changes, which can blow your schedule out of the water,” Schwalb says. “To meet the schedule, you have to go from finding several defects in tests to finding one or none,” he adds.

Intuiting what a project might need
Small groups of software developers at Intuit began PSP/TSP training in the fall of 2003, and in the following spring, the company launched pilot projects using the techniques on development work for three products, including two versions of the QuickBooks small business accounting software. Fagan says even quality-minded software engineers didn’t always immediately embrace the PSP/TSP concepts, for fear they might be a flavor-of-the-month process-improvement solution that ultimately wouldn’t be adopted.

Sensitive to this skepticism, Fagan was careful to follow the full PSP/TSP program and avoid short cuts such as compressing the training or launch phases. She also aborted one of the pilots after only a couple of months when contract developers were hired because the project proved to be much larger than anticipated. The contractors weren’t PSP/TSP trained, which goes against one of the tenants of the approach.

“Because all of the elements for success weren’t there, we chose to stop the pilot rather than try something that wasn’t going to work,” she says. “If engineers started to see things falling apart,” the commitment to PSP/TSP would have suffered, she adds.

Progress in the remaining two projects wasn’t always problem free either. At one point, the development teams began to significantly miss milestones. Fagan decided to use a TSP tool: the relaunch. “If your plan and your actual performance get too far apart, you have to stop and ask, ‘What’s going on?’” she says. “The right thing to do is create a new plan. A relaunch isn’t something you do without thought. But if there’s a systemic problem, you have to take another look at your plan.”

The first relaunch elicited “significant concern” among managers, who feared the PSP/TSP approach may not be working. But from that experience, Fagan learned to set expectations in future projects that relaunches aren’t unusual in an environment where features, resources and estimates change. “We don’t know everything” about a project during the initial launch phase, she says. With better upfront prepping about the fluidity of the software process, managers at Intuit are now more accepting of relaunches. Of course, success also helps managers accept the PSP/TSP approach. Developers in the pilots found 86 percent of their defects before the formal test phase began. As a result, the testing phase accounted for only 9 percent of the project’s total time. Non-TSP teams at Intuit have spent as much as 50 percent of their time at this stage.

Esperanto for the enterprise
PSP and TSP require long hours of training not only for software developers but also for senior managers and anyone else in the company who interacts with the programming staff. The data-gathering and planning components of the methodologies can seem tedious and even irrelevant to programmers who prefer to just dive into the coding process. Humphrey admits that the most highly productive programmers on a development staff may actually see their output decline because of PSP and TSP. But for the nonelite, the formalized approach typically boosts production by 10 to 20 percent.

And perhaps more important, it gives software developers a language for speaking with managers and the rest of the organization. A language that helps everyone negotiate more rational schedules and do better work. “There’s nothing fancy about this stuff,” Humphrey says. “It’s just the modern way to do software development.”


On ADTmag.com
Soft dev survey says people matter most
By ADT Staff

Proponents push business rules
By Stephen Swoyer

Extreme programming for a rainy day
By Matt Stephens