Tailor-Made for Applications Development
- By Alan Joch
- October 1, 2005
What’s the first thing you do when you have to write a new software program
on a tight schedule? For years, the answer has been to just start banging out
Now, a growing number of programmers in large commercial software companies,
corporate IT departments and the military are dealing with tight deadlines by
delaying the time when they write that first line of code. Instead, they spend
almost four days analyzing their resources and requirements and perhaps sparring
with senior managers to make sure that deadlines aren’t only tight, but
that they’re reasonable.
These nonconformists are devotees of Personal Software Process and Team Software
Process, two intertwined methodologies that offer formal processes for improving
software quality and managing schedules. “This isn’t rocket science
or a silver bullet,” says Eileen Fagan, software engineering process manager
for Intuit, which began using PSP and TSP early last year. “It’s
a collection of all the things you need to do to be highly successful, one project
at a time.”
Intuit declines to reveal what it spent for PSP/TSP and what the resulting
paybacks were, but it and other organizations believe the extensive data collection
associated with the methodologies point to concrete benefits.
The techniques are having an impact on deadline performance and software quality,
not only at Intuit, but also at Boeing, IBM, Microsoft, Motorola, Teradyne and
the U.S. Navy. In some cases, PSP and TSP are helping teams deliver programs
with less than one bug per 1,000 lines of code.
As time goes by
Obviously, software quality has been a driving force for developers, IT managers
and senior executives for years. However, with each passing year, as increasingly
sophisticated computer systems run everything from the most complex fighter
planes to mundane household appliances, the ripple effect of programming problems
becomes more acute. Adding to the nuisance—and danger—of quality
problems are hackers who exploit security holes in commercial software to steal
personal and financial information and feed worldwide identity theft rings.
Like most software bugs, security holes usually are fixed after an intrusion.
Now, some commercial developers and software engineering academics say enterprises
can no longer adopt the traditional break-fix approach to software development
for financial, competitive and security reasons. Instead, software developers,
working with support from their managers and senior executives outside the IT department,
need to fix code during the development process, which can also help them do
a better job of scheduling and cost-estimating large programming projects.
One way to do this is using methodologies that have become common among quality-savvy
manufacturing companies. A formal strategy for implementing these practices
is the two-pronged PSP/TSP attack being pushed by the Software Engineering Institute,
part of Carnegie Mellon University. Watts Humphrey, an SEI fellow and developer
of both PSP and TSP, has spelled out these principles in a number of books,
including Winning With Software: An Executive Strategy and PSP:
A Self-Improvement Process for Software Engineers.
PSP, first developed in the early 1990s, helps individual programmers estimate
how long it takes them to write code, document the incidence of bugs and identify
defects early in the process to avoid time consuming fixes at the end. The second
prong, TSP, arrived at the end of the 1990s and expanded PSP principles to workgroups
composed of PSP graduates.
It’s the increased adoption of that second prong among commercial and
military software engineers that offers encouraging signs that complex software
projects can be better managed. “Until there was TSP to go along with
PSP, our goals weren’t fully achievable,” says Jeff Schwalb, National
TSP lead for the U.S. military’s Naval Air Systems Command (NAVAIR, which
provides support for the maritime combat activities of the Navy and Marines).
Creating high-quality software can be a matter of life and death at NAVAIR;
programs often bolster aircraft and weapons systems.
People, not tools, matter most
best-in-class software development projects are 3.37 times faster to market
and 7.48 times cheaper than the worst, according to a new study of
"the best and worst in class projects" from Quantitative Software Management.
The costs and duration associated with being "best" or"worst" are dramatic:
the construction and test portion (the build phase) of the project for best
in class was about 3.6 months and cost $200,000, while the worst in class
took a year and rang up $1.4 million, QSM found. The total lifecycle costs
(the requirements analysis plus the build phase) are also meaningful: just
above 7.5 months and cost $300,000 total for the best in class; the worst
in class took more than 2 years at a budget-busting $2.2 million.
Four factors contributed to the results, and only one was technology-related.
The rest concerned management and technology approaches. Development teams
in the "best" category were able to control requirements change; employed
highly skilled people with good functional knowledge of the application domain;
used tooling effectively; and had project leadership.
"Controlling change is the most important element to successful development
projects," says Doug Putnam, managing partner of QSM. "An inability to adroitly
manage change can be the enemy of productivity and quality. Effective
leadership creates a culture where change is well managed by highly skilled
teams with good domain knowledge."
Although tools are always a significant factor as productivity drivers, they
ranked only third as an influencing factor for the quality of the projects,
according to QSM. There's no technology substitute for training and
management, the study concludes.
QSM surveyed 536 IT projects from 31 companies in 16 countries, including
16 industries and 9 sub-sectors, ranging from aerospace to financial IT. --ADT
ALTERNATIVE APPROACH TO BANGING OUT CODE
- Some commercial developers and software engineering academics say enterprises
can no longer adopt the traditional break-fix approach to software development
for financial, competitive and security reasons.
- Software developers need to fix code during the development process, which
can also help them do a better job of scheduling and cost-estimating large
- PSP and TSP require long hours of training not only for software developers
but also for senior managers and anyone else in the company who interacts
with the programming staff.
It’s like factory work sometimes
PSP and TSP offer an alternative to the decades-old software approach of “banging
out code and testing it,” Watts believes. “That approach leads to
re-dos. Anyone with any quality background will look at that and say that’s
fundamentally wrong,” he says.
With PSP and TSP, he applies quality principles introduced by W. Edwards Deming
and Japanese auto makers that focus on quality at the beginning of the job.
“The fundamental principle of factory work is you do it right the first
time, which makes you cost competitive because you eliminate redundant work,”
He says software developers haven’t traditionally taken this approach
because they’re often pressured by management to meet unrealistic schedules
that induce errors. “Senior managers always push completion dates until
they get enough push-back to convince them that this is the best they can expect,”
Humphrey says. “The problem is software developers haven’t known
how to push back” so they find themselves agreeing to deadlines without
first determining whether the schedules are realistic. “Basically, all
we’ve done with PSP and TSP is to show software developers how to push
back,” Humphrey says.
He set out to develop this language in 1989, and over two years, wrote 70 programs.
Along the way, “I gathered data on everything, I managed quality and I
defined processes—everything I thought should be done,” he recalls.
“And I found it worked.” Using statistics on how long it took him
to write programs and by documenting defects as he went along, Humphrey could
accurately estimate schedules and improve the quality of his programs.
He began teaching PSP courses that mirrored his experiences. Current classes
require students to write eight programs using a standard process: Make a plan,
develop a program, do a post mortem. “During those steps, students gather
data about how much time it takes them to do the work, measure how big the program
is and record all of their defects,” he explains.
Although there’s a lot of formal classroom and textbook study associated
with PSP instruction, TSP is much less theoretical and more hands-on, says NAVAIR’s
Reconciling needs with desires
TSP consists of a number of project development phases, beginning with a detailed
planning session. According to TSP gospel, the development staff first hears
a project summary from senior management that spells out the parameters of the
new software initiative, including the deadline for completing the code, the
business case for launching the effort and any details—often scant—about
specific functionality the new software should offer.
Next, the development staff sequesters itself for 3 1/2 days to compare project
goals to available resources and to the data collected by PSP methods showing
how fast individual staff members can write nearly defect-free code. Then, they
reconcile this analysis with senior management’s deadline expectations.
Out of the hundreds of teams I've seen go through this process, I've yet to have a team lose an argument, because they have a plan, lots of data and they put a lot of effort into it.
Day 4, when developers present their conclusions to management, is typically
when the fireworks fly. Rarely do a development team’s conclusions meet
the deadlines set by management, Humphrey says. But the analysis process provides
a way for the two sides to hammer out a more workable schedule, preferably with
alternatives the development staff identified in advance. For example, the programmers
might ask for more resources or cutbacks in the functionality of the proposed
“We tell the teams to do their best to do what management wants and prove
they really put their hearts into the effort,” Humphrey says. “Out
of the hundreds of teams I’ve seen go through this process, I’ve
yet to have a team lose an argument, because they have a plan, lots of data
and they put a lot of effort into it. If management argues, everyone on the
team can come back with a response. When the whole team is there and everyone’s
behind the effort, management listens.”
Although the planning phase is a key starting point to TSP, it’s not
a one-time effort. Schwalb says large projects typically incorporate multiple
planning sessions that coincide with key milestones.
Meeting schedules with fewer bugs
NAVAIR has been using PSP since 1996, after Schwalb heard a presentation by
Humphrey a year earlier. NAVAIR added TSP around 2001. The original move to PSP
was part of the organization’s overall desire to improve its software
development processes, a move that first led it to classical process improvement
techniques that included the Capability Maturity Model, a framework developed
by the SEI that spells out hundreds of development best practices.
The emphasis of CMM and its cousin CMMI (Integrated) is organization-wide process
management, not the individual and team orientation of PSP and TSP.
NAVAIR’s software managers became frustrated with CMM after spending
months creating new processes, then finding themselves conducting seemingly
endless training session to get everyone onboard. “It was just taking
too long, and people were running out of energy,” Schwalb recalls. When
he learned about PSP, however, “I made paradigm shift,” he says.
“It gave us a starting point and a way to tailor the development process.”
Within about six months of taking SEI-led PSP training courses, many of NAVAIR’s
developers began to feel comfortable collecting performance data and using it
for project management. Success rates in hitting scheduling targets soon began
to improve, Schwalb ways. Although PSP moved NAVAIR a step closer to where Schwalb
wanted to go, it wasn’t a complete answer. After going through PSP training,
many programmers said they weren’t using what they’d learned because
others in the organization weren’t similarly trained. “That was
frustrating, but nobody said stop the training, Schwalb recalls. That frustration
eased in 2000 when NAVAIR began incorporating the workgroup concepts of TSP
that the SEI was formalizing. “We saw real successes when we applied TSPin
its full version around 2001,” Schwalb says.
Planning and ongoing timetable assessments have had an impact at NAVAIR. In
the past, projects may have come in anywhere from six to nine months behind
schedule. The new PSP/TSP techniques have helped the staff hit deadlines within
one week of the original schedule, Schwalb reports. What’s more, code
quality goes hand in hand with hitting schedules. One reason schedules are more
accurate is that NAVAIR is spending less time fixing errors at the end of the
Late fixes are not only time consuming, they’re especially costly because
changes often have a ripple affect on other sections of completed code, which
then must be revised to accommodate the corrections. “When you go into
test phases, you spend lots of unplanned hours or perhaps even weeks on the
changes, which can blow your schedule out of the water,” Schwalb says.
“To meet the schedule, you have to go from finding several defects in
tests to finding one or none,” he adds.
Intuiting what a project might need
Small groups of software developers at Intuit began PSP/TSP training in the
fall of 2003, and in the following spring, the company launched pilot projects
using the techniques on development work for three products, including two versions
of the QuickBooks small business accounting software. Fagan says even quality-minded
software engineers didn’t always immediately embrace the PSP/TSP concepts,
for fear they might be a flavor-of-the-month process-improvement solution that
ultimately wouldn’t be adopted.
Sensitive to this skepticism, Fagan was careful to follow the full PSP/TSP
program and avoid short cuts such as compressing the training or launch phases.
She also aborted one of the pilots after only a couple of months when contract
developers were hired because the project proved to be much larger than anticipated.
The contractors weren’t PSP/TSP trained, which goes against one of the
tenants of the approach.
“Because all of the elements for success weren’t there, we chose
to stop the pilot rather than try something that wasn’t going to work,”
she says. “If engineers started to see things falling apart,” the
commitment to PSP/TSP would have suffered, she adds.
Progress in the remaining two projects wasn’t always problem free either.
At one point, the development teams began to significantly miss milestones.
Fagan decided to use a TSP tool: the relaunch. “If your plan and your
actual performance get too far apart, you have to stop and ask, ‘What’s
going on?’” she says. “The right thing to do is create a new
plan. A relaunch isn’t something you do without thought. But if there’s
a systemic problem, you have to take another look at your plan.”
The first relaunch elicited “significant concern” among managers,
who feared the PSP/TSP approach may not be working. But from that experience,
Fagan learned to set expectations in future projects that relaunches aren’t
unusual in an environment where features, resources and estimates change. “We
don’t know everything” about a project during the initial launch
phase, she says. With better upfront prepping about the fluidity of the software
process, managers at Intuit are now more accepting of relaunches. Of course,
success also helps managers accept the PSP/TSP approach. Developers in the pilots
found 86 percent of their defects before the formal test phase began. As a result,
the testing phase accounted for only 9 percent of the project’s total
time. Non-TSP teams at Intuit have spent as much as 50 percent of their time
at this stage.
Esperanto for the enterprise
PSP and TSP require long hours of training not only for software developers but
also for senior managers and anyone else in the company who interacts with the
programming staff. The data-gathering and planning components of the methodologies
can seem tedious and even irrelevant to programmers who prefer to just dive
into the coding process. Humphrey admits that the most highly productive programmers
on a development staff may actually see their output decline because of PSP
and TSP. But for the nonelite, the formalized approach typically boosts production
by 10 to 20 percent.
And perhaps more important, it gives software developers a language for speaking
with managers and the rest of the organization. A language that helps everyone
negotiate more rational schedules and do better work. “There’s nothing
fancy about this stuff,” Humphrey says. “It’s just the modern
way to do software development.”
Soft dev survey says people
By ADT Staff
By Stephen Swoyer
for a rainy day
By Matt Stephens