The Agile Architect

Agile Testing Heresy: Are You Testing Too Much?

Our Agile Architect commits heresy by suggesting that developers are writing too many tests. Perhaps the question should not be, "Do I have a test?" It should be, "Do I have the right test?"

• By Mark J. Balbes, Ph.D.
• May 13, 2014

It's one of the tenets of test-driven development (TDD) that you shouldn't write a single line of code without first having a test to show that it is necessary. Over the years, our ability to write tests and our zeal to test everything has led to tremendous advances in automated testing, with a lot of powerful tools that let us test in ways never done before. The question is, as with all things great and powerful, "Should we?"

The Case for Testing
The standard explanation for why we create automated tests is that by having them, we can know that our system continues to work properly even while adding new functionality or refactoring existing code. Test-driven and behavior-driven development takes this a step further by using automated tests to drive software design.

Testing methodologies and tools have matured over the years, feeding off of each other. Originally, testing was based around state. If you put the system in a specific state and ask it a question, does it give you the right answer? As TDD techniques push us to create smaller classes with a single responsibility, interactions between classes have been recognized as first-class citizens of the testing world. In other words, its not enough to create a unit test that checks an object's state. Now it's important to test that the object is interacting with other collaborating objects correctly.

Here's an example. Suppose we have a system of objects A, B and C that collaborate to provide a capability. We want to unit test A, which means we do not want to use objects B or C in our test. We only want to know that A is sending the correct messages to B and C, and that A reacts properly to messages sent to it by B and C.

In order to test these interactions, we can use a mock object library, of which there are many to choose. We can use a mock object to stand in for object B and a different mock object to stand in for object C. The mocks record how object A interacts with them and can be programmed to respond to these interactions according to the needs of the test at hand.

For example, if I want to test that A handles an error condition from B properly, rather than creating an actual B object and contorting it so it's in the desired error state, I simply create a mock and tell it to return an error message when object A interacts with it.

Mocks make it much easier to do interaction based testing, thus allowing us to create an automated unit test for every external interaction of our object under test.

The Case Against (Too Much) Testing
But here's the rub: Soon you have a system of objects that work together to create the functionality you want. But, because you've insisted on test coverage for every object in the system including how they interact with the other objects, you've now encoded the details of your implementation of the system, as expressed through object interactions, into your tests. At this point, making a change as to how these object interact becomes extremely cumbersome and inhibits ruthless refactoring.

And why did we do this? We don't really care about how the objects work together. We care that the system behaves correctly as a whole. So maybe we have too many tests.

Where Do You Draw the Line?
Here's the theoretical scenario: Suppose you want to build a calculator that adds, subtracts, multiplies and divides. You start with a simple test that shows your Calculator class can add. Then you add another test showing it can subtract. Now you start to add the ability to multiply but realize you've already coded two very similar mathematical operations and now this is a third. Furthermore, you're getting concerned about sticking to the single responsibility principle because now your Calculator class knows about adding, subtracting and how to switch between the two operations. You decide to create an abstraction for a Mathematical Operation. So you refactor your Calculator class so it only knows that it contains mathematical operations. It doesn't care what they are, just that they exist and it can ask them to do things. You then create Add and Subtract mathematical operations as classes. You perform this refactoring without adding any new tests since you aren't adding new functionality, just moving code around. During and after this refactoring, your tests showing add and subtract still work. However, now they are no longer unit tests. They are integration tests since they include the Calculator, Add and Subtract classes.

At this point, we are very tempted to add a new test to Calculator. This test would use a mock Mathematical Operation to show that the Calculator makes the correct calls to the mock under different circumstances. Most good agile developers would jump to add this test and perhaps even delete the original unit tests that have now become integration tests. My assertion is that this interaction-based test is not needed and that we should instead keep the integration tests. The integration tests are testing what we care about, namely adding and subtracting. If we ever decide to refactor the Calculator, interaction-based tests that assume the implementation details of a Mathematical Operation abstraction become problematic.

I can hear the cry of the agile now! "But if there are no interaction tests, how can I ever reuse the Calculator class and know that it is working properly?"  And this is where true agility comes in. When we created the Calculator class, we were building it for ourselves. There was no requirement for reuse outside of our application. If we decide to reuse the Calculator, e.g., for another application that wants to use different Mathematical Operations, then maintaining the contract for how the Calculator interacts with other Mathematical Operations objects becomes much more important. At this point, we would want to add these interaction based tests so we have confidence that the Calculator is not refactored in such a way that it breaks the contract with Mathematical Operations.

Final Thoughts...
You can think of your software as a complex network consisting of objects that perform processing on data and objects that move data. My current frame of mind tells me that we should always unit test objects that process data, always integration test networks of objects that work together to move data, and only some of the time unit test objects that move data. When to do which is the art of software testing.  Regardless of how far you take this, commenting out any line of code in your software should still result in a broken test somewhere, although not necessarily in the unit test for that class. It could be in an integration test that includes that class.

My final warning to you about this column: Although I do this for my personal projects I've never put the practices I'm describing into place with a product team. But I'd sure like to try!

Editor's Note: Mark's original title for this column was "Don't mock your tests. It hurts their feelings." He generously allowed me to change it after extorting this concession.