I Hate It When My Tests Pass

This post is a few years old now, so some details (or my opinions) might be out of date.
I would still love to hear your feedback in the comments below. Enjoy!

I’ve been writing a bunch of unit tests recently, and I came across an odd behavior. I was writing the tests for existing code and whenever I would write a test, as soon as it passed, I wouldn’t trust it. Nothing ever works the first time around. It simply can’t be. So I found myself repeating quite an odd ritual:

  1. Write a test.
  2. Run it. It passed. There’s no way that really works…
  3. What to do, what to do? I know! Let’s change the expected value on the asserts and see what happens!
  4. The test fails! Yay!
  5. Change it back. Now I know it works!

At the moment I realized I was doing something weird, I immediately saw all the unit tests I ever wrote flash before my eyes. I was always doing this ritual*. So I’m thinking - what can be done to give me more confidence in my tests? One solution I can think of is a somewhat exaggerated form of verbosity. Take the unittest module for example. Its assert method all take a msg argument, which is printed in case the assert aborts. Well, how about adding a success_msg argument? Imagine working on a test, setting the verbosity to a maximum and see the following:

Asserting that result_value equals 5... success!
Asserting that result_error is empty... success!
Asserting that the correct exception was thrown... success!

These messages can even be automated, to some degree. 

Does this happen to you as well? Do you have a solution to this problem?

*Actually, there is a subset of my tests in which I have very high confidence. Those are the tests I developed with TDD. This post isn’t the right scope for that, but this is one of the advantages of TDD.

Discuss this post at the comment section below.
Follow me on Twitter , Facebook or Google+

Similar Posts