The Agile Architect

Project Management Is Not Enough: 4 Crucial Agile Techniques

Why you need automated testing, continuous integration, test-driven development and -- yes -- pair programming for agile to be successful.

• By Mark J. Balbes, Ph.D.
• November 22, 2011

I was having lunch at an agile conference recently with a group of strangers. We all picked the same table randomly and started sharing our experiences. Of the five people at the table, two introduced themselves as agile consultants and scrum masters, two were developers and then there was me. Nobody knows what I am.

The developers started explaining how their team had attempted to be agile but weren't succeeding. They were on the verge of giving up and going back to their old processes that had worked better for them. The two scrum masters were extolling the virtues of scrum and, admirably, trying to figure out why this other groups' team was failing. They asked all kind of questions about how long their sprints were, what their stories looked like and other great questions about agile project management processes. It was frustrating to them because the developers answered all the questions "correctly." They had hired a scrum master and coach to train them. They were doing sprints. They were customer-focused. Everything seemed to be in place. So why were they failing?

I was chomping at the bit to ask some questions of my own but couldn't manage to break into the conversation. Eventually, the two consultants left and I got some alone time with the developers. But I had a different agenda in mind.

I asked about their automated testing framework -- they didn't have one.

I asked about their continuous integration process -- they didn't have one.

I asked if they used test-driven development -- they didn't know what that was.

I asked if they paired -- they thought I was crazy.

And I knew why they were failing.

You see, agile project management isn't enough. As a mathematician might say, it's necessary but not sufficient. Here's why:

The underlying goal of agile software development is to rapidly deliver high-quality code that meets the users' needs. The presumption, borne out through years of experience, is that software requirements change rapidly. The project must be able to keep pace with these changes or risk being obsolete before it is even delivered.

Agile project management practices go a long way to making this possible. They keep requirements flexible and allow the software requirements to change pretty much on a whim.

The catch is that the software itself has to be able to change on a whim, too. It's one thing to be able to say halfway into your project, "Our users say we're out of step with their needs. We're going to change our e-mail messaging system into a live chat messaging system." It's another thing to actually do it and still deliver on time.

Here's my list of the critical techniques required for truly successful agile development, where rapid requirements changes can be followed through with rapid development changes. As you can guess, they are the same techniques I asked the developers about in the story above. So here they are, from most important to least:

1) Automated Testing
Automated tests are nothing new to the disciplined software developer. While some may think this was invented when JUnit and other similar frameworks were created, I can attest that I was doing automated testing in C for a medical application long before that.

And though it's nothing new, it continues to amaze me how many projects are developed without their benefits. A solid bed of automated tests (unit, integration, regression, user acceptance...) means that the software can be dramatically changed without the fear that existing functionality is going to break.

2) Pair Programming
Here's the scene: two developers, one keyboard, one mouse, one large monitor, and one story for them to implement together. Can it really work?

Pair programming is perhaps the most controversial of the techniques we're discussing but it is also one of the most important. Even in my company, we continue to have differences of opinion on how much to pair program and how much to go solo. On my team, the rule is that a pair must write all production code. Individuals can perform tasks like spikes and code reviews.

Pairing gives your team depth and breadth of knowledge. It prevents knowledge silos that often lead to bloated code and many, many bugs. The members of my team change but we can always absorb new members and get them up to speed quickly. After all, they get one-on-one training from their pair. When someone leaves our team, their particular talents leave with them but the knowledge stays. 

3) Test-Driven Development
Test-driven development is an important extension of automated testing. Rather than writing automated tests after the code is written, a single test is written and then just enough production code is written to make the test pass. After the test passes, you look at the code to see if it needs to be refactored to a better design. Then you repeat the process: test, code, refactor. Slowly you build up your knowledge of what you are building, how to build it and how to prove that it is working properly. In addition, the automated tests serve as documentation for the design. Unlike UML diagrams or other written documentation, the tests are living, breathing proof that the system works.

Test-driven development is not something that is easy to do. It takes practice and a willingness to change how you go about building software. But the rewards are plentiful.

4) Continuous Integration
Continuous integration is usually thought to be an automated process using integration servers like Cruise Control or Jenkins. These servers continuously monitor your source control server (e.g. Subversion or Git) for changes, pull these changes, build the system, run all automated tests and then publish the results. While this is great, on smaller projects you don't necessarily need this overhead.

Continuous integration is really the act of a team committing their changes to the software often (e.g., multiple times per day), pulling the changes from others and then running all the tests to ensure that what they are writing has not broken anything already committed to the source code repository. In this way, everyone is working on the same baseline software.

If you think this seems obvious, let me tell you a story: Many years ago, I was working on a traditional software project. When I joined the team, everyone was committing broken code to the repository. They knew that the build only happened on Friday so they had that long to fix their code. One poor guy was in charge of the build. Every Friday, he pulled the latest code, built it and then published it out for everyone to use as a base for the next week's development.

Sounds easy, right? Only, the build didn't get finished until the following Tuesday or Wednesday because nothing ever worked. And I'm not talking about broken tests -- we didn't have a lot of tests. I'm talking about not being able to get the application to compile for days. It took me about a month, but I managed to get a build server running, train the team on how to properly commit working code (and pull down everyone else's changes) and monitor the continuous integration build server for failures.

But that isn't the story. The story is about our teammate who was in charge of the complicated back-end data analysis engine. He was working solo. (No pairing on this project.) He had no automated tests. (You can forget about test-driven development.) And he didn't pull down any changes to the software for three months!

Go ahead and read that last sentence again. For three months he worked totally isolated. And shame on us for not catching it. We ended up throwing away most of what he had done and reworking it.

Quick Quiz
You and your team are working hard toward a release date. You're following all of the practices laid out above. You've got your pairing workstations set up. You change pairs multiple times per day. Everyone writes the code using test-driven development techniques and they share the work effectively using techniques like the pairing game.

Then your boss walks in. He announces to the team that the project must be shipped early. Your six-month deadline is now three. He pulls you aside and says, "I know this is going to be tough but I figure you spend about half your time writing tests. So if you stop writing tests, we can meet the new deadline."

Do you say:

1. "Great idea! And with the time we save, we can add more features!"
2. "Darn tootin'! And we'll stop fixing the existing tests when they break!"
3. "Hmmm. I guess you've risen to your level of incompetence."

Or maybe you say something else. What would you do?

Final Thoughts...
The development techniques I describe above are crucial in order for agile to fulfill its promise of delivering high-quality software that can rapidly adapt to changing requirements. They allow the software developers to change direction and potentially make sweeping changes to the design and architecture of the software while ensuring that existing functionality doesn't break.

Nope. These Are My Final Thoughts
I'm sitting here writing this, imagining you reading it, your jaw slowly dropping, your eyes widening, and thinking to yourself, "Fat lot of good this information is going to do me. I'm working on a project that's been around for three years and we don't do any of these things! There's no way we can start now."

Fear not. All is not lost. In my next column, I'll talk about how to deal with technical debt and legacy code. Stay tuned!