Testing too much, testing too little
Several years ago, when I was just starting out in the tech world, I was really lost when it came to deciding what my automated tests should be testing. I have heard about all the great developers who swore by the need for unit tests to make their software more reliable. And then there were the arguments about complete code coverage or fuzzing tests to check every possible input. So when it came to my own projects, I was completely overwhelmed by the possible options.
Luckily, just a few weeks before this, I met someone amazing at the dinner with some friends. Fred had an incredible career, having worked on some of the most exciting problems in tech. He's also one of the few people to possess an immense technical knowledge while also being a genuinely fun person to be around. I sought his advice and the below is my perspective on his recommendations. This simple approach is something that has worked for me on my own projects, but there are many cases where a different method would be needed. So, I hope this helps you think about tests, but perhaps do not follow this if you are building aircraft safety systems at Boeing.
When starting a new project, it is clear that there are two extremes:
- not using automated testing at all and relying on some manual checks
- going all in on the code coverage, fuzzing, and TDD
If my aim is to just build something in a relatively short amount of time that works for the purpose I built it for, spending an enormous amount of time on testing seems crazy. But then again, I wouldn't want the thing to be a buggy, glitchy mess for myself or somebody else using it.
The approach I use is simple - I split the potential test areas into easy to mess up and hard to mess up. For example, as most of us know, off-by-one errors are very easy to mess up. Meanwhile, a variable assignment or an if statement checking whether a number is greater than 10 is much harder to mess up.
These areas can then be split into easy to test and hard to test. Of course, some of the areas that are hard to test might indicate a problem with our design. Thinking about this is a good thing itself, as we are now aware of the areas that might need to be re-thought.
To prioritise writing tests for your project, you can just follow a simple order:
- Easy to mess up & Easy to test
- Easy to mess up & Hard to test
- Hard to mess up & Easy to test
- Hard to mess up & Hard to test
Adding tests in this order helps getting more feedback early on, which should steer the development process away from the obvious errors. Then, as we approach the hard to mess up and hard to test, there is diminishing value for the amount of effort spent.
As many might have noticed by now, this approach is the commonly used impact-effort matrix. In my experience, changing the wording of the columns and rows makes it easier to reason about the specific trade-off between how likely a mistake is to happen and the amount of effort needed to test whether it does happen.
Now, you might argue that there are many areas that are semi-hard. That these areas might be quite challenging to categorise. In most cases, just choosing one is better than pondering for too long and not getting anything done.
If we write tests for all four splits, how is this different to just writing tests for everything? Well, we can choose to stop early on any of the four splits/quadrants. Some programs need more testing and some need less.
For example, if I build a sound board for my stream that plays silly sound effects whenever I die in a video game, I could just get away with writing tests for Easy to mess up & Easy to test areas. Meanwhile, for a slightly more serious project that you want to maintain fast progress on, you might focus on the whole Easy to mess up row.
To make a choice, consider the worst consequence of an error occurring during the runtime: the greater the consequence -> the greater the need for more tests. And if you work on anything safety critical, please close this window, as I hope you know better than to follow the advice of an anonymous blog.
In the face of the unknown, errors are inevitable, but only through failure can we glimpse new truths.