The Agile Architect

Measuring the Business Value of Projects with Agile

Measuring business value is a challenge regardless of the methodology you use. Our Agile Architect explains how agile principles and practices can be used to ask the right questions.

• By Mark J. Balbes, Ph.D.
• August 30, 2012

In my last column, I promised you that we'd discuss measuring business value this month. While I fully intend to do that, first I want to set expectations appropriately. This is a really hard problem. There is no right answer, and you can never definitively know what the true value of a project is.

Why is that, you ask? After all, this should be a pretty straight-forward problem -- a simple mathematical equation: Business Value = (money earned) – (money spent). Or is it Business Value = (money earned)/(money spent)? Umm, and then there's something about opportunity costs, right? If I'd spent the money on a different project, could I have had a greater return?

And then there are all the intangibles. What if someone on the project team doesn't want to be on the project so they decide to find another job? There's a cost to replacing them. Or the project brings in money but at the cost of your users' goodwill, e.g. forcing a paid upgrade.

So much of business value is wrapped up in the specifics of your business that it would be rather arrogant of me to say that I can tell you how to measure your business value. But I can give you some ideas of how agile techniques can inform your decisions and provide some metrics-driven approaches to measuring that value.

One way to attack a large, poorly defined problem is to break it down into more manageable pieces. We can do that with business value by asking two questions:

1. How do I measure the relative business value of features within a project?
2. How do I measure the business value of projects with respect to each other?

Let's do the easier one first.

Measuring Business Value of Features Within a Project
In previous articles, I've talked about agile estimation and story sizing. There is another kind of story sizing that can also be beneficial to track: the relative business value of each story.

I've recently been experimenting with this on my projects. The customer (i.e., the business representative on our team), not the developers, estimates the relative business value for our stories. Assignment of the business value points is as much an art as it is a science. 

With the business value points assigned, we can then watch how many of these points we accomplish each week as well as our velocity. While we expect our velocity to stay flat or increase over time as we get better at development, we expect our total business value points per week to decrease as we work on less important stories as the project progresses.

Measuring this business value burn-up through the life of the project provides multiple advantages. First, the practice itself forces the customer (or customers) to go through the exercise of looking at the stories to figure out their relative value. This brings up questions that might otherwise not be asked. For example, user interface cosmetics and usability stories tend to be ranked as less important than functionality, at least initially, by customers. Once they start using business value points as a way to prioritize stories, they realize that they want to place these low business value stories ahead of much higher scoring stories. Whether that's right or wrong, whether the points are wrong or the priorities are wrong, whether points don't absolutely determine priority, all depends on the project and the business. What is important is that the discrepancies are being pointed out and discussed.

Ultimately, tracking the burn-up of business value points allows you to see quantitatively when you are getting less return on your investment in a project. This lets you end the project when it is no longer providing enough value, rather than at some arbitrary calendar date or pre-determined dollar expenditure.

Measuring Business Value Between Projects
Measuring business value between projects is much trickier. Just as story points are not comparable between project, so too are business value points only meaningful within that project.

Well, that's not quite true, at least if you keep good project metrics. A team's historic velocity tells them how much work they can get done in a given week. If you track the man-hours spent developing a story, you can pretty quickly build up a metric that relates story points to man-hours.

O.K., I hear all you agile coaches screaming, "Story points don't equate to real hours! It's a relative measure and it's only an estimate.  Please stop your insanity before you convince management that they can hold us to these estimates."

I hear your cries and I agree with you -- in the "small." Story points are estimates, and for any given story they won't translate into an accurate estimate of man-hours. The same statement is not true in the "large," given certain caveats.

If I've tracked the man-hours spent on each story, then I can calculate the average ratio of man-hours to story points for an aggregate set of stories, e.g., for a minimal marketable feature or a small release. If the standard deviation of the average is small (i.e., the team estimates consistently) and I have averages from a lot of minimal marketable features and releases that maintain that consistency, and the team makeup stays relatively constant, then I can use that average ratio to predict how much work is left on a project in terms of real man hours. Real man-hours translate across projects and can be used to estimate the actual costs associated with a project. I do this routinely on my projects and it does work, but you have to build up the metrics first using a mature, stable team.

So that's information about the cost side of the equation. What about the actual value side?

That's going to depend on what your business actually is. For example, if you are a company that sells software products, then you may try to equate business value points to revenue based on historic metrics, just as we did for story points. Given the difficulty of partitioning a single revenue stream, e.g., sale of a product, to all the projects that contributed to it, it might be impossible for many businesses. But if you have multiple products, it could give you quantitative metrics for comparison. (Fair Warning!: This sounds like a reasonable approach to me, but I've never done it.)

On the other hand, if you're a company that builds software for internal systems, then IT may be a cost center with no profit capability at all. In this, case, business value is achieved by minimizing IT costs while maximizing enablement of the rest of the company. You would want to identify real-world metrics (e.g., man-hours spent on support calls for each system or hours spent by users to accomplish specific high-value tasks) to equate to business value points for projects for a given system.

Ultimately, the desire is to have metrics on your projects that feed business value calculations that contribute to a managed project portfolio.

Taking It Forward
Measuring business value is a tough proposition regardless of the methodologies used. Agile practices provide a natural way to follow a metrics-driven approach that is based on historic truths rather than prophetic suppositions. While I can't tell you what specific metrics are important to your business, I hope I have provided you with some food for thought on how those metrics might be used.

I'll leave you with an open question. Given a metrics-based approach, rather than attempt to analyze business value in a vacuum before a project starts, might it be better to start projects, see what the metrics say about business value relative to other projects, then fail fast if the value isn't there?