Columns

Are you agile?

• By John D. Williams
• October 1, 2001

During construction, there were more elevators on the outside of the building than the 73 that are currently inside it. Total construction time was only one year and 45 days, and it was completed ahead of schedule. Why do I mention this? Because, until we can build software systems of similar complexity in a similar time frame, I don't think it is accurate to call software development "software engineering." Software engineering remains more a goal than a reality.

Lately, there has been a lot of discussion about lightweight and agile methods as a superior approach to software development. My question is, do these methods come closer to engineering practice? I'm not sure they do, but I think they deserve a closer look. As an example of the philosophy of this approach, let's look at the values and principles found in the "Manifesto for Agile Software Development" presented at www.agilealliance.org. We start with the statement of values.

"We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

These values sound good, but are they any more than warm, fuzzy sentiments? Will they stand up in the real world? At the very least, these values reflect an understandable reaction to the difficulties organizations have experienced using heavyweight processes. This doesn't mean that undisciplined chaos is just as good—this is not what the manifesto writers suggest. Instead, the challenge seems to be finding the right balance of process and human initiative that allows software development to rapidly and consistently deliver high quality solutions. This is the heart of turning development into an engineering discipline. Does the agile approach succeed with this? Let's take a look at some of its principles.

First consider this: "Our highest priority is to satisfy the customer through early and continuous delivery of valuable software." This point is one with which many in software development would agree. Frequent and continuous releases using iterative development have great value for satisfying customers and ensuring the delivered product meets customer expectations—even if those expectations change during the course of development. With more than a decade of iterative object-oriented development behind us, we can find numerous cases in which this principle has proven its value. This principle is closely tied to another one. "Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage." Whether or not the agile methods strike the best balance between accepting changes and avoiding never-ending scope creep remains to be seen. Still, approaches that have the flexibility to accept changes even late in the development cycle are sure to help teams deliver software that meets the changing needs of a business.

Another principle that reflects agile method values is: "Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done." This sounds nice, but does it really make sense? Here we see the preference for people over process. Certainly the quality of the people on your development team will have a significant impact on development. And yes, you do need to trust the team to get the job done. Does this mean that people are more important than process?

Edwards W. Deming used to conduct a "red bead" experiment during his seminars on quality. He created a manufacturing line where the task was to produce white beads from a mixture of white and red beads. People from the audience became the workers, supervisor and quality control staff. The workers were given paddles with indentations to hold the beads. The workers dipped the paddles into a tray containing both white and red beads. Workers could manipulate the paddles any way they chose. Those who met or exceeded quota were rewarded. Those who did not were harangued and later fired. There were slogans and mottos to encourage workers. People shared their techniques to get the most white beads. What did the experiment reveal? In the end, it was clear that no one could do any better than the process allowed.

The lesson that Deming wanted people to take away from the experiment was that the process defined the lower and upper limits of how well people could perform, and no amount of sloganeering, rewards or punishments could change that. I think the same message applies to software development. Any process you pick, including none at all, will put limits on how well you can perform. Processes set the bounds of performance. No amount of trust, best intentions or group hugs will change this. The team will only be able to improve their overall performance by improving the process. Preferring people to process for software development is like preferring one side of a quarter to the other. Both are needed and, in fact, are inseparable.

Another consideration is that process and methodology are really mechanisms for risk management. You may trust your team completely, but you may need to adopt particular processes to mitigate risk. For example, you may have external risks or factors that need to be managed such as a market window or customer contract terms. There may be internal risks as well. So, yes, trust people. Give them as much freedom as you can—within the constraints of your process. However, simply trusting people to do a good job is not an acceptable substitute for engineering discipline.

Another principle of the agile approach is: "The most efficient and effective method of conveying information to and within a development team is face-to-face conversation." This certainly seems to have some validity, but there are some questions this principle seems to ignore. People may talk, but they do not always recall a conversation the same way. Documenting a conversation helps clarify issues when memory grows fuzzy. Just thinking of the immediate team is also too small a view. What about maintenance? Typically, the team that supports a system is different from the one that developed it. How will decisions and assumptions be passed on to others if conversations are not documented? Conversation is necessary, but it is not a substitute for documentation. I think this is one area where the agile emphasis is inadequate to meet the needs of software engineering in the enterprise. The agile approach seems to focus on small group dynamics, that is, the team. The amount of formality needed for a team to function smoothly is less than that needed to replicate that behavior consistently across an enterprise. The agile approach may work well for a team, but how do you replicate that across 1,000 people around the world? It is unrealistic to expect every team to have the same caliber of people. Only a more formal process can replicate capability consistently.

So, does an agile approach move us closer to true software engineering? Can you build the Empire State Building this way? I think there is real benefit in the values and principles espoused in the manifesto. However, in some ways, its focus is too parochial. The emphasis of the agile approach is on the team of people and their skills. It appears to ignore enterprise issues. Also, the manifesto has a flavor of people versus process. This is not a useful dichotomy. Neither people nor process is the answer by itself. Both are needed. The challenge is finding the right balance. Rather than thinking about heavyweight or lightweight processes, we should think about just enough process. I submit that the right process is the one that accommodates the constraints (requirements) of the business and the common business goals. These constraints and goals are more than those at the project level. Process must accommodate overall business constraints and goals. At the project level, there are typically fewer constraints, and processes can be more flexible. Higher-level constraints may not seem relevant to a project, but they are necessary for overall alignment to business goals and maximizing the value of the project to the business. They also make it possible to replicate capability consistently across the enterprise. Will agile methods find a way to address the larger enterprise issues and move us closer to a true software engineering practice? It's difficult to say. It will be interesting to see what fruit agile methods bear over the next few years.