Nick Carroll

The quickest way to get code into production is not to write tests. [Waits for reader to stop shouting "That is blasphemy!".] Now hear me out, this statement is true based on the assumption that the code will run as expected. However, you cannot know with certainty that your code will run as expected without testing it. Therefore you can write code and deploy it into production quickly by not writing tests, but it does not mean that the code will be defect free. As a result you run the risk of spending an indefinite amount of time fixing defects post release. This takes your time away from developing new features that add more business value.

The opposite is to spend more time upfront writing tests. This results in a longer period of development effort to get code into production, but you have more confidence that your code will run as expected. Which leaves you more time post release to work on new functionality that will add business value, instead of being swamped with fixing defects.

Now you have two contrasting views. The first is that you can quickly deploy code into production by not writing tests, but you have almost no confidence that the code will work as expected. The second is to write tests to improve confidence, but it will take longer to realise any business value. These views provide a sliding scale for software quality, at one end you have no tests and thus almost no quality assurance. At the other end you have unit tests, integration tests, functional tests, and more tests, all of which provide you with a high degree of quality. In a perfect world you would opt for the latter. But we don’t live in a perfect world. Everything costs money. Clients have a finite amount of money to spend on projects that deliver business value, and they want the best bang for their buck.

In the Agile world business value is proportional to the development team’s velocity. The higher the velocity the more business value you’ll eventually get. On a recent project we tested everything, and I mean everything. We had unit tests, integration tests, functional tests, and end-to-end tests. We did TDD, so most of these tests were written before a single line of code was written. Our test coverage was in the 90 percentile range. We had a high degree of confidence that what went into production was bullet proof. This is not to say that there were no defects, there were some defects, but they were all resolved in a timely manner. This kind of leads me to the main point of this post, you can test the hell out of something but it doesn’t guarantee that it will lead to defect free software. In other words you can have 100% test coverage but your software may not deliver what the customer wants. Or you simply didn’t cover scenarios in which your software will be used.

On this particular project the iteration manager (Agile project manager) would always state “we should be going faster”. He was right. We had a team of highly capable developers, and some of the things we were doing was not rocket science, but the business domain was quite complex. We were not experts in the business domain, and I believe we compensated by writing a lot more tests than we needed to. Through the power of hindsight I can say this, but on the project the team was geared towards writing tests. In our retrospectives the question of “how can we go faster?” was always raised. There were many useful suggestions, but nobody dared to utter the “T” word. I was always thinking it but didn’t have the balls to say it. Is this a symptom of group think? That as a team or organisation for that matter, we believe that writing tests are good, and so writing a lot more tests must therefore be better. It hardly seems Lean.

The signs that our tests were getting in the way are now glaringly clear. We had a nightly build that ran a large suite of acceptance tests and with a large set of data. Most mornings were spent figuring out why the nightly build failed. Sometimes it was due to an integration server going down, sometimes it was due to a genuine bug being exposed by testing with a large data set. The problem here is that there were a number of false positives that took time to verify and as a result took time away from working on a story.

Another sign was that our pre-commit builds took about 15 minutes. These builds included a decent size set of Selenium based functional tests. Now 15 minutes is a hell of a lot of time to be waiting around for your build to finish. To get an idea of what it was like take this xkcd comic and replace the word “compiling” with “building”. It not only holds you up, but the rest of the team as well. We had this practice of sequential check-ins, in that you have to be holding the “check-in chicken” before you were allowed to check in any code. This queued up check-ins in that you had to wait until the chicken holder had finished their build and checked in, before you were allowed to build and check in your code. It sucked even more when there were two or three other pairs in front of you. Most of the time you would just keep working, but this prevented you from doing small atomic commits, which resulted in larger commits and conflict problems.

Tests do slow you down, and it is pure ignorance to believe otherwise. It wasn’t until my most recent project where my first instinct was to test the hell out of something. The team lead on this project asked “why do you want to write that acceptance tests?”. My initial guarded response was “are you kidding me?”. This led to a massive debate over the value of writing tests. This debate was mostly between the team lead and another developer. I was too absorbed in the concept of not having to write tests for everything. Picture a cloudy sky opening up and a ray of sunshine falling down upon me. That was how I felt when I found another person that challenged the status quo of test the mofo out of everything. The default stance that we take is test everything. Usually this is done with little thought as to whether extensive testing is really required or not. Whether the client asks for it or not they are going to get a thoroughly tested system.

This leads me to the main purpose of this post, to challenge this default stance of test everything thoroughly, no matter how long it takes. It came about through the debate that we had about testing, which led to the team lead putting us in the client’s shoes. If hypothetically you had $100,000 and needed to have your website built, would you be happy that you got one piece of functionality and about five lots of tests for that functionality? Probably not. You might be happier if you instead received three pieces of functionality and a test for each functionality. I am over trivialising this, but you should get the point. Economist would call this disposition of utility gained from having extra tests in exchange for functionality marginal utility.

As in economics I believe the law of diminishing marginal utility also applies to writing tests. In other words there comes a point where the utility of writing tests detracts from the overall goal of delivering business value. This leads me to the JET acronym that I coined which stands for Just Enough Testing. It is loosely based on the concept of JeOS or Just enough Operating System. JeOS allows you to start with a stripped down operating system to run in virtual machines, then add additional services to the OS to suit the environment in which it will be used. I would like to use JeOS as an anology as to how JET should be used. JET is a stripped down testing strategy that you must adopt, then build on this strategy to suit the project that you are on. Simply adopting the stance of test everything is not a strategy that should be employed on every project. The clients’ demands will be different in all cases, and their marginal utility for business value over tests will vary.

My JET approach is to write unit tests for designing the interactions between the class under test and its collaborators. Integration tests mostly for end-to-end testing, but with the awareness of not going nuts here. Finally having a minimal set of functional tests using Selenium to cover typical workflows through a web application for example.

Adjusting JET to your project is easy. If you have the luxury of a QA for example, then get them to spend more time in writing functional tests so that all regression tests become fully automated. If your site is quite simple on the front end then your minimal set of functional tests should give you enough confidence that your application will work as expected. If however your back end is more complex, then spend a bit more time writing integration tests in that area. JET is not about writing less tests. It is about realising that projects have constraints and you need to work effectively within those constraints, and that means thinking about whether you really need to have that functional test or not, because maybe just having an integration test will give you enough confidence as to how your code will work.

In all honesty JET is just a cool acronym that relates quite nicely to velocity. Whether JET takes off (I couldn’t help myself with that pun) or not is completely left to you to decide. I’m not the only one that thinks it is perfectly ok to compromise when writing tests. Kent Beck also thinks so in his article Where, Oh Where To Test. Kent Beck comes to the conclusion in that article that a pragmatic programmer should decide what to test and how much testing is required based on three criteria: cost; reliability; and stability. From the article: “The testing strategy that delivers the needed confidence at the least cost should win.” In other words you should do just enough testing.

I know there are plenty of test zealots out there. I’ve worked with some of them, and know their counter arguments quite intimately, and they are only sound arguments in a perfect world where a client has an infinite budget for a project. For those people I have provided the pretty pictures of jet aircraft above. For the rest I hope you find the post to be an interesting read.