In-Depth

'Extreme' Method Simplifies Development Puzzle

• By John K. Waters
• January 1, 2001

New lightweight XP methodology stresses teamwork, testing, planning and simplicity; uses real-world experience as design patterns for a new generation of software developers.

The words "extreme programming" conjure up visions of thrill-seeking programmers bungee jumping into projects without a plan, of caffeine cowboys jamming code through the night with nary a document to guide them, and of daredevil developers working without a net. In fact, the principles and practices that comprise this emerging software development methodology, known to its relatively small but expanding circle of devotees as XP, is, in many ways, about as controlled as you can get.

According to Uttam Narsu, senior analyst at Cambridge, Mass.-based Giga Information Group, XP is what is known as a lightweight software development methodology. Lightweight methodologies have few rules and practices, all of which are fairly easy to follow. In contrast, heavyweight methodologies, which most software developers currently embrace, involve elaborate rules, extensive up-front planning and piles of ongoing documentation — what Narsu calls the "artifacts of the process."

"[XP] is reminiscent of the patterns movement," noted Narsu, "which comes out of the Smalltalk world. It embraces the notion of being able to abstract real-world experiences of software architects into design patterns that guide people who don't necessarily have years of experience. It basically says, 'Look, lots of people have been developing software for many, many years. They have evolved a series of practices that work. Let's use those practices.'"

Ward Cunningham, one of the fathers of XP, agrees: "Extreme Programming is a lot of simple little things. It's lots of things that have been done before, some of which have even been discredited. We have reconstructed a collection of practices that support each other in a way that is startling."

Cunningham is a computer consultant well known for his work in object-oriented programming practices. He is one of the "Three Extremos" whose work and thinking helped produce this methodology. His Web site, Wiki Wiki Web, has become the de facto online home of XP.

Computer consultant Kent Beck is the Extremo generally credited with assembling XP practices for the first time and naming the methodology. He did it almost on the fly in 1996 when he was brought in as a consultant on a floundering project at Daimler-Chrysler. Chrysler wanted to replace a group of disparate payroll apps with a single appli-cation. Officially called the Chrysler Comprehensive Compensation project, it is known in XP lore as "C3." Beck took over the project, dumped everything that had been done before and started from scratch.

"In the course of two or three weeks of interacting with the Chrysler people, I made it all up," said Beck. "If you look at it, there's really nothing new there. I guess you could say that it was putting the pieces together. I would literally stand up in front of [a white board] with a marker in my hand and not know what I was going to say. It was an exciting time."

Beck describes XP as a "programming-centric" methodology. "It says that the thing we as programmers do that adds value to the business is write code," he explained. "Everything else is window dressing, and we should do as little of everything else as possible. Documentation, design, formal review, separate QA; it's all a waste of our time. [XPers] do what we do very intensely, but we don't do a bunch of the other stuff."

Testing: The heartbeat of XP

Testing has long been the ugly stepchild of software development, but Cunningham describes it as the "heartbeat" of the XP methodology.

"Instead of doing big design up front, we write tests up front," he noted. "Every day before you write code, you write a test to see whether that code works. It's the practice that prevents chaos. It is through constant testing that an XP project yields a very high level of certainty that the program is working. It gives developers a kind of freedom you don't find in other methodologies."

XPers test like maniacs throughout the development process. They perform unit tests for each class as they code. All unit tests in the system must be running at 100 percent before any developer releases his or her code. The C3 project reportedly had more than 1,300 unit tests.

XPers also perform functional tests of each feature defined by a user story. Unit tests check the behavior of a single class; functional tests check the system from end to end. On XP projects, developers build a continually growing collection of functional tests, which they run automatically, usually overnight. XP proponents recommend that functional tests be run daily.

"The idea," said Beck, "is that the developers are responsible for proving to their customers that the code works correctly; it's not the customer's job to prove that the code is broken. From a customer's standpoint, it's all about transparency, risk reduction and the ability to adapt to change. And the customer gets real measures of what is actually going on in the project, not someone's opinion. They get demonstrated progress. Fifty percent done of an XP contract means that 50 percent of the features are 100 percent complete."

Of user stories and pair programming

A cornerstone of the XP process is the user story. A user story is a short (about three sentences), plain English description of what the customer wants the software to do, written by the customer. Each feature of the application is defined by an individual user story. They are printed out or, more often, hand-written on cards, sort of like a Hollywood scriptwriter's storyboard. From these user stories, XPers estimate how long a project will take, establish priorities and develop functional tests.

User stories take the place of the large requirements documents typical of heavyweight methodologies, and they differ strikingly from traditional requirements specs in their level of detail; they are short and to the point. When it is time to implement the story, the developer and the customer sit down together and expand on the original description.

According to Beck, a typical XP project of six to 12 months calls for between 50 and 100 user stories. If there are too few stories, it becomes difficult to define priorities; if there are too many, the whole thing becomes unwieldy. Individual user stories are broken down into engineering tasks. If it looks like an engineering task is going to take more than a week to complete, that task is broken down further.

In addition to providing the user stories, customers play an ongoing role in the XP process. For all intents and purposes, the customers are part of the team. They must be readily available to answer questions, resolve disputes and set small-scale priorities.

The XP methodology emphasizes teamwork, and works best with small teams of between two and 10 programmers. The whole team owns the software, and anyone can find him or herself working on any part of it. XP takes the team approach to, well, extremes in a practice known as pair programming. In an XP project, programmers double up, two to a workstation. XP proponents insist that two programmers working side-by-side are more than twice as productive as one for a given task.

"Pair programming provokes talk between programmers about programming," explained Cunningham. "As they struggle with the lack of language to describe what they're doing, a vocabulary emerges. Pretty soon a vocabulary develops, a language that is about the act of programming. And that's something that people have just plain done without — to their detriment. This is an experience that people have just not shared. But because we can talk about it, it forms in our minds a much better mental model of programming."

According to Ron Jeffries (the third Extremo), many coders are at first resistant to the idea of pair programming. "But once they try it," said Jeffries, "most of them really like it. I'm not comfortable with the idea of pairing two junior programmers, but you can put them together in just about any other combination. Senior programmers — real superstars — find that working with an inexperienced programmer makes them better. And the junior programmers get better much faster when they work with an experienced [person]."

Jeffries worked with Beck on the C3 project, becoming XP's first "coach." In an XP project, the coach oversees the project and makes sure that it remains extreme. The coach maintains the project vision and communicates it to the other team members. The coach also keeps individuals on track, egos in check and even acts as a mentor when appropriate.

Planning the XP way

Their aversion to monolithic processes notwithstanding, XPers do plan. They get past some of the pitfalls inherent in heavyweight methodologies with a process called "the Planning Game" — which actually takes the form of a game, with goals, playing pieces, players and moves.

The basic playing pieces are the user stories, which are written on cards. The players are the customers and the developers, who get together and move the cards around on a large table. The basic goal of the planning game is to create a release plan that specifies which user stories are going to be implemented for each release and the dates for those releases.

Once the release plan is set, customers and developers get together for iteration planning meetings, which are held at the beginning of each iteration to produce that iteration's plan of programming tasks. Customers choose from the set of user stories that will be implemented during the next iteration. These selected stories are then translated into individual programming tasks that will be implemented during the iteration to complete the stories.

XPers plan individual iterations in detail, but not until just before each iteration begins. (For the complete "rules" of the planning game, point your browser to http://c2.com/cgi/wiki?PlanningGame.)

In its pursuit of a clean and spare overall design, the XP methodology also employs something called a "spike solution," or just plain "spike." A spike is a very simple program that XPers write to explore potential solutions to tough technical or design problems. According to Cunningham, Beck coined the term: "I would ask Kent [Beck], 'What's the rock-bottom simplest thing we can program that will convince us we're on the right track?' That question takes you outside the complexities within the existing code, which is probably distracting you from the essence of the requirement. The idea is to write the smallest possible code that could be said to perform a function independent of the existing mechanism."

In the XP methodology, spikes serve primarily to help a programmer work through a problem; once the problem is sorted out, most spike solutions are thrown away.

According to Cunningham, it was the advent of object-oriented programming that served as the impetus for the development of XP. "Everybody knew that OO had a lot to offer," he said, "but people had a lot of trouble with it. Objects are about modeling; modeling is about making judgments. With algorithms and data structures you can sort of get by any which way. But to get objects to work, things have to be written well."

So, how do you take the plunge into an XP project? There are basically three ways to get started: provide your development team with immersion training before you begin; hire a coach to work with you during the course of the project; or as Beck put it, "Just rearrange the furniture and get going."

One of the few places — maybe the only place — to go for XP training is a company called Object Mentor Inc. in Green Oaks, Ill. The company offers mentoring, training and outsourced development services. Both Beck and Jeffries teach through this company.

If you plan to jump in on your own, Beck recommends starting with the Planning Game. "If you don't have the stories, or at least the beginning of the stories, you risk wasting lots of time solving problems that aren't problems. I ask this question a hundred times of a new XP team: 'What story are you working on?' Generally the answer is, 'Well, none, it just sort of seemed this stuff needed some work.'"

Cunningham suggests taking on one XP practice at a time to introduce your firm to the methodology. Unit testing may have the most impact on the project, he said, but he agrees that the Planning Game is invaluable. Unit testing has the advantage of being relatively easy to adopt. Pair programming can be a good place to start, too, he noted, but if your people are uncomfortable with it and you have to harp at them, they will probably miss the point.

One of the fastest ways to ramp up an XP process is to hire an experienced coach. "By the time you get to iteration four," Cunningham said, "you'll know what you're talking about. It seems to take that long to get the kinks out."

Is XP ready for prime time?

Critics argue that XP and other lightweight methodologies differ little from the style of programming that formal methodologies replaced. It is a step back, they say, to the bad old days of unstructured development. Giga's Narsu dismisses such charges.

"I don't really think any of the XP people are saying, 'Gee let's throw away design and just code,'" he said. "Many of the critics who make that kind of argument don't really understand XP."

XP is not throwing out all the rules, explained Jeffries; it is simplifying methodologies and eliminating practices that hinder the development process, while hanging onto rules that work. "If you look at the situation," he said, "almost no one is really following the heavy methodologies anyway. I'm seeing people who seem to be moving instinctively toward XP." Jeffries describes XP as "a backlash against heavy methodologies." The lightweight nature of the XP methodology, added Beck, is a reaction to document-oriented software processes that saw coding as little more than a clerical task, something that was done after all the interesting work was finished.

"Many people in development are observing that [following heavy methodologies] just doesn't work," said Beck. "XP is clean and concise, and it allows programmers to be creative and productive while remaining organized and focused."

But XP does have its shortcomings; chief among them, said Giga's Narsu, is its customer participation requirement. "In many of the real-world projects that I've dealt with, the reaction is not terribly good among the stakeholders. These folks have a lot to do, and the notion that they should be available to clear up any problems with a user story ... well, it's not going to work in many situations."

Another issue Narsu is concerned with is the inherent limitation of the pair programming practice in distributed development environments. "Can you really do pair programming when one programmer is in Paris, Texas, and the other one is in Paris, France?" he asked. "Even with the good collaborative tools available today, is it really the same as sitting next to your partner?"

Beck sees the methodology's tendency to expose organizational weaknesses as a potential downside of XP. "If your marketing folks are telling you what to do every two weeks, it's obvious that they're clueless. Lots of people react to that by huddling up and not communicating with [the developers]. The proper XP response is to make it obvious that those decisions are not being made. You should say that no programming projects can go forward until [those decisions] are [made]. The responsibility has to go somewhere; on an XP project, it goes where it belongs."

Giga's Narsu worries that XP may simply be too new to have all the kinks worked out. "Keep in mind that we don't have much experience with this methodology," he said. "This is an evolving methodology. Is it ripe and ready? It's hard to say at this point."

There are no hard statistics on the number of XP projects underway, but most of those interviewed for this story believe that there are about 100 — a number Jeffries describes as "insignificant in the grand scheme of things, but surprising, given how resistant most people are to change."

Marty Thorne's Portland, Ore.-based company, the Client Server Group, is currently utilizing the principles of XP on a project for Pacific Gas & Electric. He said that the "extreme" moniker is a bit of a misnomer. "It's called 'extreme' because XP sort of drowns out any other practice," explained Thorne. "An extreme project is very stable and almost sedate. Because of the principles, you can move forward fearlessly. You can change code at will. You're designing, but you're doing it incrementally. And you only put in what you need."

Who wants to go to extremes?

Anecdotally at least, there seems to be a lot of interest in XP these days. XP sessions at the Software Development (SD) 2000 conference held in San Jose, Calif., in March were jammed with attendees. Marc Pardi, director of the conference, said that the enthusiasm he saw at the XP sessions reminded him of SD's first Java sessions. "[XP] has an almost 'cultish' following," he said. "And what's driving that is the pressure on developers to get corporate e-commerce sites up and running now. Even their corporate apps are driven by the Web. The developers are the unsung heroes in this new economy. These guys need the tools, and XP certainly seems to be one of them."

Although Beck takes credit for assembling the initial core principles of the methodology during his involvement with C3, he allows that the process grew from other methodologies, and that it is evolving in the hands of developers.

"It's sure not my baby any more," he said.

As for the name, Beck defends it vigorously. "It was exactly the right name for it," he said, "because it takes what we do to extremes: everybody unit tests; we build and test eight to 10 times a day, we code review constantly, we iterate on a one- or two-week cycle. XP pushes its practices to the limit."