The Agile Architect

True Quality Assurance in an Agile World

Quality Assurance is not just about testing a system. The role of QA on an agile team is much greater than that, elevating the QA engineer to a god among men.

• By Mark J. Balbes, Ph.D.
• October 31, 2012

I admit it: I'm thinking this is going to be an easy column to write because it came to me, whole hog, earlier today as I found myself espousing the virtues of quality assurance (QA) in a team retrospective. Let's see if that pans out.

In the retrospective, the team was discussing the imminent loss of their QA engineer to another project. We were discussing how we could fill the role, given that he was not going to be replaced by an outside person. One of the other team members suggested that the departing QA engineer describe his role on the team so that we could identify who would be the best fit.

He began his description rather timidly, "Well, it's pretty simple. Basically, when the stories are done, I test them to make sure they meet the acceptance criteria. Since the developers write the Cukes (Cukes are tests written in Cucumber, an automated user acceptance testing framework.) I run them to make sure they pass. Then I do some manual testing to make sure everything works."

Of course, he had fallen into a common trap for QA on development projects. He had resigned himself to the role of quality control (QC). QC is simply the act of testing something once it has been built to make sure it meets quality standards. It does nothing to ensure that those quality standards are going to be met by the production process.

I couldn't let this stand. I launched into a description of the role of true quality assurance. And while I described this role in terms of an agile project, it's applicable to any kind of project.

"First of all," I began, "QA starts at the very beginning of the software development process, with story creation."  Quality assurance is responsible for making sure that the output of the software development process is a quality product. Of course it has to start with stories. But it's not QA's job to define stories. That's for the customer. So what does QA do?

For stories, QA's role is to ensure that the stories are written following quality standards. In a previous article, "Defining Your Product with Agile Stories," I discussed the canonical forms for stories ("As a <some kind of user>, I want to <do something> so that I can <get value> ") and for acceptance criteria ("Given a system in a specified state, When something happens, Then the system responds in a specified way"). This is a great start, but it's obviously not enough to ensure that any stories written are easy to understand, provide real value to the end user and are small enough to be implemented in a reasonable amount of time (e.g., a day).

The QA role, then, is to write acceptance criteria for stories and acceptance criteria. Yes, I really did say that QA should be writing acceptance criteria for acceptance criteria. Sounds crazy, eh?

In other words, for a given project, what are the quality evaluation parameters for a story or set of stories? The examples I gave above are pretty much universal, but every project will have additional criteria specific to that project. It is QA's role to ensure that the team develops that list of criteria and applies it to each story.

Here's a small sample of some questions/considerations we apply to each story on one of my projects:

• Is the behavior different when the system is offline?
• Does the user see the same thing after restarting the system?
• Are aspects of this story only relevant inside the context of a mission?
• Do we have too many acceptance criteria for a single story such that it turns into an epic?

The above criteria can be applied to stories, to group of stories (sometimes called "Minimal Marketable Features") and to the feature as a whole. While it's not necessarily the responsibility for QA to write stories, although on some agile teams they do, they are certainly central to the story writing and review process.

Once the stories are written and the team has ensured that they meet the quality acceptance criteria, the team can now estimate the level of effort for the stories. And who better to ensure that these estimations occur and are done in a consistent fashion than QA? Sure, the agile coach is involved too. They'll have to work together on this one, but the coach has a different point of view. 

During estimation meetings, QA is looking at the event at multiple levels. At the simplest level, QA is making sure that estimates take into account the level of effort for testing. Most of the time, the testing effort scales relative to the development effort, so there don't have to be specific story points allocated to testing. However, there are some stories where testing outweighs development. There are even stories that are pure testing, e.g., "We think the system already does this, but want to write some automated tests around it to make sure."

At a higher level, QA is ensuring that the story estimation process is being consistently followed so that meaningful, quantitative metrics can be tracked based on the estimates. In addition, QA may be looking at ways to refine the entire story definition and estimation process to improve the effectiveness of the team.

Now that the stories are written and estimated, they are ready to be developed. Of course, QA is on-hand to work with the developers on the automated unit, integration, performance and user acceptance tests. Depending on the maturity of the development team, they may need more or less help with these efforts. On my MFK project, we have weekly field tests where we take the equipment outside and walk around the city testing it in realistic conditions. Our QA engineers (Yes, you can have more than one!) are responsible for creating the field test scenarios each week to exercise our new capabilities.

In addition, QA tends to have a better working knowledge of the system than developers, so they may act as a customer proxy when the customer is not available. QA may also act as the conscience of the continuous integration build server, making sure the team stays on top of broken builds.

And finally, we get to the end of the line, quality control. QA is still here, making sure that everything works as required. They review the acceptance criteria of each story, ensuring that the automated user acceptance tests (UAT), as implemented, meet the intent of the original acceptance criteria. They add additional UATs as needed. They run all the automated tests. They perform manual testing including experimental and performance testing and then determine whether these should be turned into automated tests.

Afterward they get the distinct honor of declaring any given story/minimal marketable feature done.

But that's not all...There are QA processes around a release, too. But I think that's going to have to wait for another column.

A Frame of Mind...
You may be thinking that my description of the QA role matches that of the agile coach. And you are right that there are a lot of similarities. However, the agile coach is shepherding the entire agile development team with a different frame of mind. The agile coach is constantly asking the question, "How do I help the team become more effective and more agile?" Quality assurance is constantly asking the question, "How do I ensure that our team puts out a quality product?"

And, yeah, I was right. This was an easy column to write.