Waterfalling Forward to Agile

• By Kathleen Richards
• July 1, 2007

Companies that develop apps on the .NET Framework and J2EE platforms are more likely to use Agile processes, a new survey shows. Agile development, which is loosely based on principles outlined in the Agile Manifesto penned in 2001, are dev processes designed to get developers and business stakeholders to work together to adapt quickly to changing requirements. eXtreme Programming, Scrum and other Agile methodologies support short cycles of development that result in working software at the end of each iteration.

For development teams, this means more collaboration and face-to-face communication, unit and automated acceptance testing during the development cycle, and responsibility for "deliverables" every few days or weeks. The payoff, according to proponents, is a "self-organizing" development team, competitive advantages and higher quality software.

Although these dev methodologies, like AJAX, have been around for years, enterprises of all sizes are starting to move to Agile processes. Agile adoption increased in 2006 in large enterprises across the board, reports Forrester Research Inc. One-sixth of 1,078 North American and European companies surveyed in September by Forrester indicated that their dev teams are using Agile processes. Business services (25 percent) and financial and insurance companies (23 percent) reported the highest adoption rates.

Done, Done
Ryan Martens, founder and chief technology officer at Rally Software Development Corp., an Agile software and services company in Boulder, Colo., sees repeated patterns when his company helps people move to Agile processes. "When you're trying to get a team spun up to Agile, the first level of maturity is about getting a team into a smooth flow," he advises, "so that they deliver some increment of functionality every two or four weeks." When the development and the acceptance testing are accomplished, it's what he calls, a "done, done."

Once you get the repeatable increments coming out of the development environment, you're going to be a lot more efficient, asserts Martens. This first step towards Agile adoption makes it easier for teams to estimate what needs to be done, and to "steer," because they close out development every few weeks.

"Agile practices are easy to misunderstand and misapply," says Carey Schwaber, senior analyst at Forrester Research. Many companies that Schwaber encounters run into serious issues when the team reverts to a mini-waterfall methodology. "Letting phases creep into the Agile iteration is one of the most common pitfalls," she warns. "It's really hard to root out the waterfall mindset and get concurrency of activities."

Moving Test Forward
Test-driven development is another challenge for developers who are used to working on long development cycles with a series of drops to QA. "To try to get to 'done, done,'" says Martens, "it means you need to be able to collaborate, write and run your acceptance test in a two-week cycle." This is really hard for people who are used to building acceptance tests after they've got the code. Two techniques can ease the transition here. Developers can start to write a lot more unit tests, and they can develop acceptance tests in frameworks that don't require having the software done, such as a GUI framework.

[click image for larger view]

Agile Adoption and .NET

Bottom-Up Adoption
Agile needs to be adopted from the bottom up on a team-by-team basis, asserts Martens. "The team itself is the basic organizing mechanism for Agile," he says. In a self-organizing team, the software development manager tries to get the obstacles out of the way for the team. However, if you take the self-organizing concept too far too quickly, you risk ending up with the "self-empowered team" and cowboy coding, warns Martens: "'I don't need to plan, I don't need to write docs,' and it reverts toward chaos," he says.

Best practices for Agile include executive sponsorship, advises Schwaber, because Agile adoption often involves more changes than dev managers foresee. "You'll need someone with the authority to help you get these changes made," she says, "especially changes that span organization boundaries."

Finally, be patient, says Schwaber: "Agile adoption is a fundamental transformation of how development is done and how teams work from the top to the bottom. It isn't the kind of thing that happens overnight."