In-Depth

Process fortifies software projects against failure

• By Buff Colchagoff
• October 1, 2004

For most developers, process is a bitter pill to swallow. But employing standard best practices in a repeatable way is critical to software organizations that want to build quality solutions delivered on time, within budget and to the requirements of the users.

The key to implementing process as an effective medicine against software project failure is determining the right dose for your organization. By customizing the process to fit the needs of your enterprise, you can effectively close the gap between stakeholders’ requirements and what developers produce. The next hurdle is to gain acceptance from developers, which requires modifying behavior across the organization.

Most developers consider process a bitter pill to swallow. In fact, many consider it a placebo that placates managers, but does nothing to help produce software. When we look at successful projects, however, process plays a critical role. Is there a magical, Mary Poppins-esque spoonful of sugar that helps the process medicine go down? Maybe, but let’s examine whether developers have some justifiable grounds for their distaste.

If we want to change developers’ attitudes, we need to understand their concerns about process. Developing software for more than 20 years, I’ve found that it’s common for developers to view their organization’s software development process as onerous, inefficient and unnecessary.

When speaking about process, developers conjure up the idea of three-ring binders containing volumes of required forms. I recently worked with a developer on filling out the required form that documented each software change for a quarterly release. The problem was the documentation’s level of detail: The form required a table with entries for every SQL statement that changed. It’s easy to see how a developer might call this onerous.

Sequential processes like the waterfall model can be inefficient. With it, the workflow can be understood and managed easily because each task follows in a certain, unchanging order. However, development is more efficient when it is collaborative and concurrent with other activities. For example, doing architectural tasks during requirements gathering raises technical issues early and avoids costly rework later.

Developers see some process tasks as unnecessary. For example, one organization required all technical artifacts be reviewed by the quality management (QM) team at the end of every phase. Unfortunately, the QM team did not understand the technologies involved. So instead of a review of the technical content, the meetings would focus on document formats. In one case, the significant review discussion debated whether a document’s section should use a table instead of a bulleted list. Developers walk away from these reviews frustrated and questioning their value.

Let’s be honest. Developers are notorious for only wanting to code. Most would be happy to skip planning, reviewing, talking to users and testing to sit at their computers and code. To many developers, everything else is a nuisance or doesn’t seem important to producing software.

Could they have a point?
To some extent, developers’ gripes are justified. Many organizations have outdated processes that don’t reflect current industry best practices. Viewed individually, each inefficiency in a process may not seem significant, but over the entire project life cycle they can become weighty baggage.

Worse, when official processes are viewed as too cumbersome or useless, developers and even project managers just don’t implement them. This presents a huge risk to the organization’s ability to produce quality software on time -- and those three-ring binders just become dusty and old. An unused process is worse than no process at all because it seems to prove that process adds no value.

Software production’s illness requires medicine
A common industry belief is that 70% of IT projects fail. This notion is a little dated and imprecise. The 2003 Standish CHAOS research suggests 15% of IT projects failed and 51% were “challenged.” These results show that IT is hobbled by low project success rates. Of the top 10 identified success factors, most are within the domain of process, including user involvement, project management, scoping, methodology and requirements management.

Sometimes, we view process as the collection of stuff in the three-ring binders. However, I’d like to explore another view of process -- one that thinks of it as not just the medicine, but the prescription.

In this view, the major goal of process within an organization is to apply best practices --those techniques and activities that have been proven to improve efficiency and quality in software production. By following a prescriptive process, an organization sees repeatable results. It’s hard to argue against using process this way -- employing best practices in day-to-day software production.

In fact, implementing best practices is critical to repeating project successes. Producing software with teams is an inherently complex task. To repeatedly produce software that is reliable, on time, within budget and that meets or exceeds users’ expectations requires dynamic workflows that foster collaboration, reveal risks and ensure consistent quality.

Unfortunately, many organizational processes aren’t designed this way. So what can be done?

Designing processes
Curing software project ailments is similar to visiting a doctor. In the same way a doctor prescribes the most effective medicine for a patient’s needs, companies need to design their prescriptive processes using best practices that match their organization’s unique needs.

There are four key practices that are critical in designing any software development process:

1. Develop iteratively. A process that uses iterative development will have a number of iterations through the life cycle. Unlike the waterfall methodology, each iteration will have a set of tasks involving the disciplines of requirements, analysis, design, coding, test and deployment. The amount of work within each discipline varies depending on where the iteration is within the life cycle. The main idea is to focus on addressing the highest project risks early. For example, early iterations focus on architecture. This is not the same as a design phase. Instead, the developers produce and test a software executable or release designed to address the architectural risks in the project. In a classic waterfall, the architecture is not tested until the test phase.

2. Manage requirements. It’s common wisdom that stating all requirements for a large-scale program is practically impossible. Yet many processes assume it can be done. A better approach is to have the process account for changes in requirements. This is not to say that anything goes until deployment. What we want is a systematic approach that not only captures requirements, but manages the agreement on requirement changes between the development team and project stakeholders.

Efficient processes leverage the iterative approach by focusing on risky requirement tasks early. For example, a common goal of the first iteration is to set the scope of the project, going deep enough to uncover significant requirements, but leaving less-significant requirement details for later iterations.

3. Manage change. Change management is critical for organizations building high-performance software teams. As tasking becomes more concurrent, and more releases and environment configurations are involved, the complexity of managing all these pieces increases dramatically. What may have worked in the past will not suffice. An effective process must include a well-defined approach to managing the changes in software, documents, environments, change requests and releases, as well as the ability to track these changes.

4. Continuously verify quality. Verifying quality becomes more difficult in an iterative framework. A non-iterative process usually has completed deliverables due at specific times. In iterative development, documents and functionality progressively mature through the life cycle, so quality must be verified at multiple points for some deliverables. This step-wise verification becomes critical to realizing the final goal.

This iterative attention to quality can also help to accelerate development times. A practice I’ve seen successfully employed on development projects is weekly quality builds. Each week, an integration build is created and tested, and defects are logged. This raises defects very early in the cycle. Developers can fix bugs before the next quality build, greatly reducing the time to do system testing and defect cycles.

Fitting organizational needs
The biggest key to designing an effective process is determining the correct dose of process medicine. It’s easy to make a comprehensively large process that blankets all risks, but that is terribly inefficient. The trick is to apply the smallest dose of medicine that is still efficacious. Albert Einstein put it another way: “Everything should be made as simple as possible, but no simpler.”

How do you know what must be done in your process? Unfortunately, there’s no quick answer. If we look at the goals of the organization, we’ll be able to identify the risks that might prevent us from successfully achieving those goals. The process should be fitted to address the significant risks.

Let’s look at an example. One organization was designing its process and asked the question, “What level of design work is needed for our organization?” The organization had been using Macromedia’s ColdFusion, a mature architecture framework, for some time and the application projects were similar to previously built projects. From an architectural perspective, the risk in new development was low vs. that of an organization tackling a complicated J2EE project for the first time. So their process required a lower level of design detail.

This example raises another point. One project can have very different risks than another based on the technologies used, the stakeholders involved and the scope of the system. So there must be a way for individual projects to customize the organization’s process to meet its needs.

Just as a doctor writes a prescription for each individual, we need to write a prescriptive process for each project -- one that accounts for the project’s risks and determines the practices and the minimum doses needed.

The challenge
Once an organization has the right process, how do you get developers (and the rest of the development team) to use it? What I’ve seen work repeatedly is a set of techniques for collaborative, incremental change.

Overcoming the existing momentum within an organization takes more than hiring a single process expert. A team experienced with implementing best practices on similar software development projects is most effective at this type of change. This team engages the organization in a couple of ways:

Lead by example. Developers will follow a new practice when they see it working. Driving change is impossible without showing developers how these new practices are beneficial. My firm uses player coaches, an expert within a discipline who is injected into a project to do heavy lifting and mentoring. We follow a teaching hospital method for training residents: “See it, do it, teach it.” This technique instructs the team, gives them confidence and evangelizes the new methods.

Pilot projects. Early in the adoption curve, the team uses pilot projects that serve a number of purposes. First, they are an example for future projects, proving that the process works and providing useful templates, examples and optimizations. Second, they provide a breeding ground for change agents. The project team trained during the pilot sees how well the process works. They are then inserted into other projects as zealots and experts, accelerating the rate of change.

Incremental change. Many efforts have failed because they’ve tried to change an entire organization’s behavior overnight. Changes should be made incrementally, with the highest risk and most significant issues addressed early. The priorities are worked out with the developers and other staff in a collaborative environment. This helps to create authorship within the development teams. It’s also important to have a plan for these changes across iterations. Developers like to know what changes are coming and when. Having a reliable plan builds confidence in the changes as they are implemented.

Results-driven with open reviews. To create buy-in, development teams should agree on the expected results of the new process. Then, at significant milestones, they should conduct open reviews of the results. At this point, they may readjust expectations for results for the next milestone. Open reviews increase process improvement and create a collaborative environment where everyone chooses to use the process.

Use tools where they add value. We’re talking process, but tools can be used to add value to process changes. For example, requirements management tools can enhance requirements analysis. Creating a requirements model rather than a set of documents enables traceability analysis that would be much more arduous without tools.

Tools are not silver bullets and they do not necessarily imply process change. I have seen many organizations that believe by simply using tools, they are using the process. Unfortunately, many are using the new tools in old ways.

Developers are pragmatic. They will gladly take their medicine, but they want to see it work. If the medicine is effective, and the prescription is appropriate for the task, developers will accept it. The techniques outlined here merely help to ease the medicine down.

A true high-performance software development organization requires more than just a good process and developers on board. Organizational change is needed to sustain consistent improvement.

Please see the following related stories: “What is a best practice?” by Buff Colchagoff

“Does my process need a face-lift?” by Buff Colchagoff