Test cases are a controversial topic in the testing community – for a large part thanks to testing guru James Bach, who has been arguing strongly against the value of test cases.
At the bottom of it, my views are not that different from Mr. Bach’s, but I would still like to moderate the anti-test-case view. I think test cases serve some important purposes.
First off, it is important to underline the difference between a test and a test case. One of Mr. Bach’s pet peeves is that running test cases is not testing, it is checking. I agree – to a point.
My main argument in favor of using test cases is that the human brain is notoriously adept at fooling itself. (Manual) test cases can be an effective method for reducing the effect of availability bias, making sure you cover areas you would otherwise overlook or dismiss as less important. Indeed, the very process of creating test cases can be a way to force yourself to reduce cognitive bias, by forcing your brain to slow down. (On a related note, I strongly recommend for any tester or other person whose profession requires critical thinking to read Daniel Kahneman’s excellent book on cognitive bias, Thinking, Fast and Slow.)
Test cases can also be a good tool for communication. A good test suite can be an excellent way for a fresh tester on the team to learn about the intricacies of an application or non-obvious business requirements of a feature.
Another point in favor of test cases is the sad fact that we live in an imperfect world. While I in principle agree with the idea that anyone who does testing for a living should be awesome, and that a product should never ship without having been tested by these awesome individuals until they are satisfied, the truth is that not every tester is awesome (yet), and not every product release has an available herd of awesome testers, with deep knowledge of both the business and the application in addition to well-developed software testing skills and enough time to apply those skills before deadline.
There are applications and situations where shipping without going through awesome testing by awesome testers is unforgivable. If you are developing an internet banking service, a government self-service platform for tax returns, or the software that controls a medical robot for surgery, there is no excuse for not using top skilled test engineers. But to be fair, the vast majority of software does not fall into this category. Budgets are short and deadlines are shorter. If you have a well-designed test suite created by (or at least reviewed by) good testers, you are better off getting your first line to run those tests, hopefully with the aid of a skilled tester covering the most critical paths, than you would be if you had to ship at the same time and just had the one tester. Even the most amazing testing skill in the world doesn’t always grant you the time to go both broad and deep before the deadline bell rings.
Writing test cases is always a tradeoff, balancing speed, accuracy and maintainability. A test case does not exist in a void – it must be designed with the target audience in mind, knowing how familiar the person running the test will be with the application and particular feature. Sometimes it will be efficient to give just the most brief cues about functions to trigger and things to remember to verify. Other times, the efficient path is to explain step by step how to build up the state that should be checked. The key point for the skilled test designer is to include just enough information for the test runner to know how to check those details that are more important than they may seem.
However, and this is the crucial point: Having a test case in front of you is never an excuse to shut off your brain. Every professional tester should know, and every non-tester who is asked to help out with running test cases should be explained and frequently reminded, that a test case is not a test. The test happens in your brain, as you look at what happens when you perform the steps in the test case, and make a sound, human judgement of whether this is a good result.
Also, though I am an advocate for using test cases, for the reasons explained above, and particularly the one about cognitive bias, it should be clear that I do not in any way recommend only using test cases. As all other approaches to testing, a test suite is a method that should never be used alone – at least not if you want to claim that the product has been well tested.