In-Depth

Agile Breaks on Through to the Other Side

• By Stephen Swoyer
• March 1, 2005

Talking Points
EXTREME PROGRAMMING

• Agile methods, and XP in particular, can be a tough sell, although they don't have to be.
  
• Agile disciplines such as XP champion a Whole Team approach in which line-of-business reps participate as team members from the outset.
  
• Shoppers can choose from XP, Crystal Clear, and FDD for starters, or explore agile project management methodologies including Scrum and DSDM.
  
• Proponents say that any successful agile effort, whether XP, Crystal Clear, FDD, or another approach, must include support from management, preferably at least two levels up.

For many developers, software development is a meticulously planned, onerously documented, and frequently lifeblood-draining enterprise. It's not uncommon, after all, for them to spend months planning, documenting, and meeting-with line-of-business, marketing, support, and other personnel before actually sitting down to write code. Not surprisingly, developers sometimes find themselves starting the process all over because they need to change the plan. Is it any wonder coding seems like an often enervating, frequently unfulfilling, and, increasingly, downright Sisyphusean task? This was doubtlessly what motivated programming visionary Kent Beck when he co-wrote with Cynthia Andres Extreme Programming Explained: Embrace Change. Beck's book championed a different paradigm-an "agile" approach to software development-that eschews the documentation-intensive, process-centric view that's often characteristic of enterprise software projects. Thanks to the prominence of Beck's eXtreme Programming (XP) methodology, interest in agile methods has surged. Coding, many programmers have since discovered, can be creative and stimulating, even in enterprise environments.

There's one catch
Six years on, business management philosophies have become even more risk-averse, with companies increasingly focused on control-intensive initiatives including regulatory compliance, or process-centric efforts such as ISO or Capability Maturity Model Index (CMMI) achievement. In this kind of corporate climate, agile methods, and XP in particular, can be a tough sell, although they don't have to be. Fact is, there's a growing body of agile-in-the-enterprise success stories, especially in large, multi-national companies with traditions of innovation.

Take Brad Appleton, a software developer with one of the world's largest manufacturers of electronics and communications technologies. Agile was anything but an easy pitch to make in Appleton's organization, but by carefully insinuating agile methods and practices where it made sense, Appleton and his programming peers were able to mount a putsch of sorts. "Initially, the agile approach was a tough sell to management, he says. "It wasn't what folks were used to doing. It didn't produce the same set of expected deliverables, and the usual metrics did not apply," he acknowledges.

One year later, Appleton is part of a hybridized XP team. What's more, his company has proposed breakthrough methods for peer review, software productivity, and defect reduction that are based-drum roll, please-on agile concepts such as pair programming.

"Now agile methods have been officially blessed by sector-wide process [or] quality policies as one of the sanctioned software development process [and] lifecycle methods for new projects to select from," he concludes.

Managing risk
Probably the best way to get a feel for how difficult of a sell agile can be is to look at how things are frequently done on the other side of the fence.

With this in mind, let's look at the gestation of a software project at a large, Fortune 500 company, in this case, at XYZ Corp., a pseudonym for the prominent network services company on which this account is based.

XYZ has its own variation on the classic waterfall model, which we'll call the software development process. Like classic waterfall, SDP describes a multi-stage process for producing software: Define, discover, design, develop, and deploy. Simply by virtue of its structure and process-centric pacing, SDP, like most waterfall-based project models, places a much greater emphasis on planning and documentation than on nuts-and-bolts programming.

A typical project scenario might involve a request from one of XYZ's business units for changes in an app that powers a customer billing system. This request is forwarded to XYZ's requirements analysis and design team, which has one job: Coming up with best-guess estimates for project costs. In this case, the RAD team computes costs based on a custom metric, let's call it level of effort, that's supposed to translate into projected hours of developer time. If the LOE estimate is low enough, the RAD team passes the request on to IT, which must come up with a lower estimate of its own.

If the cost of the project can be justified, the developer team finally sits down with the line-of-business customer. It's at this point the team and customer hash out the app's requirements in the form of a contract. Once the requirements are finalized and the contract set, XYZ's developers typically won't interact with the line-of-business customer again until the app is finished and ready for testing.

By the time the developer team starts coding, several weeks or months will have passed, and it's possible the customer's requirements will have changed, too. Unfortunately, there's not much anyone can do: Making changes to the plan, even minor ones, means generating more documents, attending more meetings, and haggling once more over projected developer person-hours or project delivery dates. Not surprisingly, says a production-support veteran of XYZ, this sometimes results in software that does little to solve business problems, even if it does satisfy the application requirements set forth in the contract.

It also leads to a kind of cover-your-own-posterior thinking among developers and business representatives alike.

"I once wrote a program that did exactly what the document said-and that turned out to be a bad thing," he concludes. "But, I had a document that said, 'Looky here, this is what you wanted, and this is what you got.'"

Bizarro programming
Agile disciplines such as XP turn this process on its head. XP, for example, champions a Whole Team approach in which customers, i.e., line-of-business representatives, participate as team members from the outset. XP doesn't expect token participation from customers, either: In the Whole Team approach, customers meet with programmers on a daily, weekly, or bi-weekly basis. The idea is to give them as much input as possible into the features, functionality, and performance of the final product.

XP is called eXtreme Programming for a reason. With the exception of Alistair Cockburn's Crystal Clear (which is a departure, in part, from XP methods and practices), eXtreme Programming is probably the least process-oriented of any of the agile disciplines, if by "process" one means a precisely defined activity. An agile approach such as feature driven development, on the other hand, may look, to a casual observer, anyway, surprisingly conventional: It stresses code ownership, code review, domain modeling, and documentation.

In FDD, as in XP, Crystal Clear, and other agile paradigms, the goal is to minimize the productivity-sapping effects of process and bureaucracy. This, advocates say, is the raison d'etre of agile programming in general. "What agile development methods say is you want to do just enough stuff other than writing code to let you write the right code," explains Roy Miller, author of Managing Software for Growth. "You don't want to clutter up the process with lots of formality and documents and meetings and such. You want to focus on creating the software that people want."

There are agile disciplines for just about everyone. In fact, it's a veritable buyer's market: Shoppers can choose from XP, Crystal Clear, and FDD for starters, or explore agile project management methodologies such as Scrum and DSDM. There are countless idiosyncratic offshoots, too.

Methods vary drastically from paradigm to paradigm, of course, but, as a general rule, agile approaches tend to emphasize customer interaction; pair-partnering among developers, who work in a room and share workstations; free-form brainstorming; rapid coding with test-driven development and frequent refactoring; and, of course, planning-albeit of a different kind. "You plan for a little bit," Miller says. "You write code for a week. You show the code to real users. They say, 'That's not what we meant at all.' Since you've only worked on it for a week, you can still change direction without losing too much."

XP is the entrÈe into agile for many programmers, and it's probably the most widely practiced of the many agile disciplines. Of course, even XP proponents concede many of the people who do XP aren't really doing textbook XP.

Take Bill Pyritz, a programming veteran with a large manufacturer of networking and telecommunications solutions, who says he and some of his colleagues are using a hybridized version of XP that's compatible with his organization's management approach. "I have been able to use agile methods in my own work with good success," Pyritz says.

"I develop [a] test first and run automated unit tests as part of the build process, along with continuous integration and task-based planning based on small steps," he says. Lasse Koskela, a programmer with the Finnish division of a U.S.-based management consultancy and global outsourcing giant, reports a similar experience.

"The use of agile methods obviously varies from project to project, so I'm not claiming that every [company] project is using XP, for example," Koskela says. "What I'm saying is that a growing number of project managers, technical architects, and software developers working for [the company] are starting to see the benefits of highly iterative and incremental development."

The point, argues author and XP luminary Ron Jeffries, is that agile methods work, regardless of what name they're grouped under. "There's a huge amount of evidence and practice around things like brainstorming, group effect, and so on," Jeffries argues, citing, for example, a recent study conducted by the University of Michigan, which found programmers who work collaboratively in a war-room atmosphere can be up to two times more productive than colleagues laboring in conventional office cubicles. This, says Jeffries, is what agile programming is all about.

A search for the elusive 40-hour work week
Many coders take up agile programming after a conversion experience of sorts. The typical agile story arc involves a coder burned out on programming for business customers and ready to throw in the towel. That's when he or she picks up an XP book, for example, and gets religion. Call it the come-to-agile moment.

That's what happened to Bob Schatz, vice president of product development with project management specialist Primavera Software. When he first started at Primavera, Schatz says, that company managed its software projects a lot like everyone else. "It was traditional waterfall," he recalls. "Everybody of course thought we must have everything nailed down because we had a [project management] tool, but that obviously wasn't the case. There were a lot of inefficiencies, a lot of bad communication within the organization, between development and marketing and support. Even in the department I was in, everything was already silo-ed, so designers were just throwing stuff over the wall and walking away."

Not surprisingly, morale among Primavera's programmers was almost nonexistent. "It seemed like every release we did was the same result. Three months of hell and then at the end, people were driven to exhaustion level," Schatz comments. "After 23 years of seeing this, I just thought something is wrong here. I just can't keep doing this. Something's going to change."

Change, and perhaps salvation, came from the Scrum project management methodology, which complemented some of the reforms Schatz had already instituted. "I was just looking at some iterative methods, and I stumbled across Scrum, and I thought, 'This would be really good for us,'" he explains.

From there, Schatz went straight to the source, arranging for Scrum pioneer Ken Schwaber to assess Primavera's corporate culture and meet with company executives. The agile agenda faced resistance at first, but Schatz says he and Schwaber used a can't-miss strategy to build backing among Primavera's C-level executives.

"In this company, it was a risk," Schatz says. "We sell project management software, and people think that we probably promote the waterfall, because that's how people think of our product. So I didn't run around saying 'I got a new process!' I just told the stakeholders, 'We're going to do something different on this, we're doing to do things more iteratively,'" he explains. The idea was to introduce Scrum project management and other agile methods gradually, so stakeholders could see the results with each improvement.

"At the first sprint review, [the business stakeholders] were just floored," he says. "They could not believe the amount of work we got done in a month. We built the buy-in by doing it and showing it, and we got some very early wins."

The rest is one for the books. "Our quality levels are through the roof, and the team productivity is just amazing," Schatz says, who is now implementing XP along with Scrum. "And we have not worked an hour of overtime or a weekend day over the last few years. But we've been putting out releases that have more features and functions than ever."

Schatz obviously is an advocate of agile methods, but he's also a realist. He says an organization often has to be in pain before it's ready to take a chance on a large-scale transformation to XP, Scrum, and other agile disciplines. "For you to do it at that level, it has to be because you're in a lot of pain-you need to do something," he explains. "Start small, with a pilot project-maybe 20 or 30 people. Don't start with 1,000 people, because you won't be able to do it.'"

Bring on the pain
For better or worse, not all organizations are in this much pain. And for would-be agile practitioners in many enterprise environments, slow and steady is the name of the game. "Management did not buy into the agile concept," says George Dinwiddle, a former programmer with a prominent Internet name registry service. "Management pushed a less rigorous, ad hoc approach to software development," he says. "That approach was to throw a bunch of warm bodies at the problem with firm delivery dates and veiled threats."

Nevertheless, Dinwiddle says, he was able to introduce practices including test-driven development, frequent refactoring, and pair-programming, albeit on a limited basis: "My immediate manager was somewhat supportive," he explains. "There was limited growth of support within the development hierarchy as success was seen."

While seeding the programming pits with agile practices and methods is a good start, XP proponent Jeffries says that any successful agile effort, whether XP, Crystal Clear, FDD, or another approach, must include support from management, preferably at least two levels up. And although there are a number of ways to make one's case to management, Jeffries and other experts note, the single best way is to give the suits working software.

"I see executives focusing on investment and risk," says Jim Highsmith, an agile pioneer and Scrum advocate who heads up The Cutter Consortium's Agile Project Management Advisory Service. "So if you give them a choice between one method in which you hand them documents periodically to indicate progress, or another in which you hand them working software to demonstrate progress, it's a no-brainer," Highsmith adds. "There's an assumption that the greatest risk is in defining the requirements, and for many, many projects, that's not a true assumption. In fact, for leading-edge products, it's the technology that's the greatest risk."

Even when XP and other agile practices have executive backing, business leaders don't always have reasonable expectations about what can be accomplished nor of how it should be accomplished.

Michael Spayd, a former CMM process assessor-cum-independent-consultant who works with XP and other agile methods, was on board at Qwest Communications several years ago when that company's CIO imposed an XP-based development model on Qwest's 5,000 programmers. For many agile proponents, this sounds like a dream scenario. In practice, Spayd says, it had its ups and downs. "This was a situation where Qwest went from CMM to XP because of a change in CIO, which is a really drastic kind of change," he says. "In a situation like this, executive backing is a necessary, but not by itself sufficient, condition [for success]," he says.

Looking back, Spayd, an unabashed agile booster, says Qwest learned several lessons about doing XP in a large organization. As to be expected, some programming teams thrived in the XP setting, but others, perhaps too steeped in old methodologies, wore themselves out trying to deliver software in 90 days. Moreover, some line-of-business customers didn't know what to do in an XP setting, while others, especially those that had a clear vision for what they wanted their applications to look like, really thrived.

In at least one case, Spayd says, XP's documentation-lite approach hurt it with a key stake holding constituency-middle managers-who weren't involved in the offsite XP training sessions for programmers. Elsewhere, only about 30 percent (or less) of all project teams experienced intensive XP training, and so one result was that many programmers merely paid lip service to XP methods and terms while going about their business as usual.

In the end, Qwest, like some of the other XP adopters featured in this article, embraced its own, hybridized version of XP, largely, Spayd says, because it determined that full-on, textbook XP didn't scale well across its environment.

Management makes the mandate
For most coders, agile programming is something they take up of their own initiative, usually by picking up a book about XP, Scrum, Crystal Clear, or another agile discipline. This is changing, however, as for some programmers, an entrÈe into agile methods comes by virtue of a mandate from management, or a new job with a new company.

That's what happened to Colin Putney, a programmer with a Mass.-based provider of network performance management solutions. "The team I belong to was originally conceived as an XP team, but after clashes with other parts of the company, XP was dropped," Putney explains.

Putney's case is a little different, of course: He works with Smalltalk, which he says naturally lends itself to many agile principles. So even though his fellow developers were forced to abandon their experiment with textbook XP, they retained the use of agile methods and practices in the midst of a process-centric business organization. "The attempt to introduce XP was more about how the team interfaces with the rest of the company, and even though it failed, we're still fairly agile, conforming to the official process when interfacing with other parts of the company," he concludes.

Sidebar: Documentation that's barely sufficient makes the grade