The quickest way to get code into production is not to write tests. [Waits for reader to stop shouting "That is blasphemy!".] Now hear me out, this statement is true based on the assumption that the code will run as expected. However, you cannot know with certainty that your code will run as expected without testing it. Therefore you can write code and deploy it into production quickly by not writing tests, but it does not mean that the code will be defect free. As a result you run the risk of spending an indefinite amount of time fixing defects post release. This takes your time away from developing new features that add more business value.
The opposite is to spend more time upfront writing tests. This results in a longer period of development effort to get code into production, but you have more confidence that your code will run as expected. Which leaves you more time post release to work on new functionality that will add business value, instead of being swamped with fixing defects.
Now you have two contrasting views. The first is that you can quickly deploy code into production by not writing tests, but you have almost no confidence that the code will work as expected. The second is to write tests to improve confidence, but it will take longer to realise any business value. These views provide a sliding scale for software quality, at one end you have no tests and thus almost no quality assurance. At the other end you have unit tests, integration tests, functional tests, and more tests, all of which provide you with a high degree of quality. In a perfect world you would opt for the latter. But we don’t live in a perfect world. Everything costs money. Clients have a finite amount of money to spend on projects that deliver business value, and they want the best bang for their buck.
In the Agile world business value is proportional to the development team’s velocity. The higher the velocity the more business value you’ll eventually get. On a recent project we tested everything, and I mean everything. We had unit tests, integration tests, functional tests, and end-to-end tests. We did TDD, so most of these tests were written before a single line of code was written. Our test coverage was in the 90 percentile range. We had a high degree of confidence that what went into production was bullet proof. This is not to say that there were no defects, there were some defects, but they were all resolved in a timely manner. This kind of leads me to the main point of this post, you can test the hell out of something but it doesn’t guarantee that it will lead to defect free software. In other words you can have 100% test coverage but your software may not deliver what the customer wants. Or you simply didn’t cover scenarios in which your software will be used.
On this particular project the iteration manager (Agile project manager) would always state “we should be going faster”. He was right. We had a team of highly capable developers, and some of the things we were doing was not rocket science, but the business domain was quite complex. We were not experts in the business domain, and I believe we compensated by writing a lot more tests than we needed to. Through the power of hindsight I can say this, but on the project the team was geared towards writing tests. In our retrospectives the question of “how can we go faster?” was always raised. There were many useful suggestions, but nobody dared to utter the “T” word. I was always thinking it but didn’t have the balls to say it. Is this a symptom of group think? That as a team or organisation for that matter, we believe that writing tests are good, and so writing a lot more tests must therefore be better. It hardly seems Lean.
The signs that our tests were getting in the way are now glaringly clear. We had a nightly build that ran a large suite of acceptance tests and with a large set of data. Most mornings were spent figuring out why the nightly build failed. Sometimes it was due to an integration server going down, sometimes it was due to a genuine bug being exposed by testing with a large data set. The problem here is that there were a number of false positives that took time to verify and as a result took time away from working on a story.
Another sign was that our pre-commit builds took about 15 minutes. These builds included a decent size set of Selenium based functional tests. Now 15 minutes is a hell of a lot of time to be waiting around for your build to finish. To get an idea of what it was like take this xkcd comic and replace the word “compiling” with “building”. It not only holds you up, but the rest of the team as well. We had this practice of sequential check-ins, in that you have to be holding the “check-in chicken” before you were allowed to check in any code. This queued up check-ins in that you had to wait until the chicken holder had finished their build and checked in, before you were allowed to build and check in your code. It sucked even more when there were two or three other pairs in front of you. Most of the time you would just keep working, but this prevented you from doing small atomic commits, which resulted in larger commits and conflict problems.
Tests do slow you down, and it is pure ignorance to believe otherwise. It wasn’t until my most recent project where my first instinct was to test the hell out of something. The team lead on this project asked “why do you want to write that acceptance tests?”. My initial guarded response was “are you kidding me?”. This led to a massive debate over the value of writing tests. This debate was mostly between the team lead and another developer. I was too absorbed in the concept of not having to write tests for everything. Picture a cloudy sky opening up and a ray of sunshine falling down upon me. That was how I felt when I found another person that challenged the status quo of test the mofo out of everything. The default stance that we take is test everything. Usually this is done with little thought as to whether extensive testing is really required or not. Whether the client asks for it or not they are going to get a thoroughly tested system.
This leads me to the main purpose of this post, to challenge this default stance of test everything thoroughly, no matter how long it takes. It came about through the debate that we had about testing, which led to the team lead putting us in the client’s shoes. If hypothetically you had $100,000 and needed to have your website built, would you be happy that you got one piece of functionality and about five lots of tests for that functionality? Probably not. You might be happier if you instead received three pieces of functionality and a test for each functionality. I am over trivialising this, but you should get the point. Economist would call this disposition of utility gained from having extra tests in exchange for functionality marginal utility.
As in economics I believe the law of diminishing marginal utility also applies to writing tests. In other words there comes a point where the utility of writing tests detracts from the overall goal of delivering business value. This leads me to the JET acronym that I coined which stands for Just Enough Testing. It is loosely based on the concept of JeOS or Just enough Operating System. JeOS allows you to start with a stripped down operating system to run in virtual machines, then add additional services to the OS to suit the environment in which it will be used. I would like to use JeOS as an anology as to how JET should be used. JET is a stripped down testing strategy that you must adopt, then build on this strategy to suit the project that you are on. Simply adopting the stance of test everything is not a strategy that should be employed on every project. The clients’ demands will be different in all cases, and their marginal utility for business value over tests will vary.
My JET approach is to write unit tests for designing the interactions between the class under test and its collaborators. Integration tests mostly for end-to-end testing, but with the awareness of not going nuts here. Finally having a minimal set of functional tests using Selenium to cover typical workflows through a web application for example.
Adjusting JET to your project is easy. If you have the luxury of a QA for example, then get them to spend more time in writing functional tests so that all regression tests become fully automated. If your site is quite simple on the front end then your minimal set of functional tests should give you enough confidence that your application will work as expected. If however your back end is more complex, then spend a bit more time writing integration tests in that area. JET is not about writing less tests. It is about realising that projects have constraints and you need to work effectively within those constraints, and that means thinking about whether you really need to have that functional test or not, because maybe just having an integration test will give you enough confidence as to how your code will work.
In all honesty JET is just a cool acronym that relates quite nicely to velocity. Whether JET takes off (I couldn’t help myself with that pun) or not is completely left to you to decide. I’m not the only one that thinks it is perfectly ok to compromise when writing tests. Kent Beck also thinks so in his article Where, Oh Where To Test. Kent Beck comes to the conclusion in that article that a pragmatic programmer should decide what to test and how much testing is required based on three criteria: cost; reliability; and stability. From the article: “The testing strategy that delivers the needed confidence at the least cost should win.” In other words you should do just enough testing.
I know there are plenty of test zealots out there. I’ve worked with some of them, and know their counter arguments quite intimately, and they are only sound arguments in a perfect world where a client has an infinite budget for a project. For those people I have provided the pretty pictures of jet aircraft above. For the rest I hope you find the post to be an interesting read.
Thank you so much for writing this article. I was actually itching to write a similar article on my blog but feel like I would have attracted the wrath of the TATFT-priests.
Really it comes down to the ideal software-engineering solution (test everything) against a practical economics. It’s all about opportunity cost: spending extra time writing tests has a benefit (ensured code quality, and with diminishing marginal utility as you say!) but it also has a cost – the forgone opportunity of developing new features.
This can be a very delicate balance and the correct decision here rests quite a bit on the client and expectations.
Also one thing to keep in mind is that opportunity cost is not always cash. In other words if you spend 5 hours writing tests @ $100/hr, the cost of those tests is likely NOT $500 an hour… in start-up companies for example developing new features is valued at more than $100/hr for 5 hours by the very fact that said money is actually spent on the features. So talking about opportunity cost in terms of developer labor time is usually not directly relevant.
In any case I may still write an article more about the economics of this (I actually have a degree in economics) but you’ve captured the essence of the business trade-offs of testing that I’ve thought about for a while.
Still in almost all cases I believe testing at least some is the right decision, but those advocating 99% test coverage all the time should at least be equipped with economic arguments for and against their decision. Such usually isn’t the case.
I agree, one thing i do is always ask myself “What is the value in writing this test ?” What feedback is this test giving me?
Also I find that starting with an end to end functional test, keeps me focused, and I write just enough smaller scale unit tests to till i get the larger scale functional test passing. I repeat the cycle, stepping back now and then to look at the overall picture of development. so far in my experience i found this very useful.
I’ve always had a similar opinion. Unfortunately, it’s very difficult to measure, so the cost / benefit is hard to determine and explain, other than gut feel and experience.
I offer one point though.. I’d say most of the production bugs found on the project you mention could be pointed back to code that was not written by a pair, but nothing to do with how much it was tested :-p
Darren, point taken. I’ve always been a lousy pair.
You are right it does come down to gut feel and experience. Working out exactly how much testing you need to do is difficult to quantify and even qualify. However, getting this out in the open is worth the discussion. Maybe somebody out there will have the answer.
So you spend most of the post arguing that writing zero tests takes less time than writing infinite tests. Then in the last few paragraphs you come out and say that there is a magic third way to do it. Just write as many tests as you need to get the job done.
I’m glad you came to that conclusion. I was rooting for you the whole time.
Darren – Very true it is hard to measure or quantify. Even so, I think it’s important that people are aware of correct *reasoning* when approaching important decisions like this. In many cases the reasoning or decision-making process is more important than the variables themselves. I find that most people aren’t even aware of the tradeoffs or don’t approach these issues with a cost-benefit/marginal value outlook.
Have you looked at Behaviour-Driven Development (BDD), which has focus on verifiable business value and just enough analysis/design/testing…
My boss sent me this article this morning because he just presented this idea to the test team yesterday. I have a great test team. We do a thorough job and it’s because we test everything that is needed to be tested (even though everyone in our unit especially business disagrees with our method). We do run a lot of regression testing. Now we understand that some tests aren’t necessary and we understand that even being the best test team this division has ever had, we’re going to miss a few scenarios and there will definitely be bugs we didn’t catch in testing. We understand that whole heartedly.
I mean, currently, I’m testing a fix to a database script I made “GOLD” a month ago and that’s because as thorough as I thought I was, I skipped one test. I integrated regression testing with smoke testing because I was under a time crunch. Instead of regression testing the entire application because after the 3rd trial of testing, 90% of the application ran beautifully without error, I decided to just generate a couple of reports from each menu item and call it good (13 reports out of 25). Little did I or the DBA or the new developer knew that a particular index was hard coded into a particular report I skipped because I took the “JET” approach.
So to me the JET approach sounds like a cop off to not do what your job entitles you to do…test. They hired you as a tester so that they can get products that have the least amount of problems when going live. The JET method is just half-assed testing and the result of that is more cost to do repairs you could’ve avoided if you ran “inifnite” tests. Either way, the testers get blamed for the failures. And regardless of how I feel about this approach, I’m going to have to make it work somehow.
The Mad Tester, I am not a tester, I am a developer that practices Test Driven Development within Agile teams. As a developer I write a test case first, then write code to make the test pass. This allows me to automate my test suite from the get go.
What I wrote above was some of the day to day decisions I would need to consider as a developer. I would write unit tests for all new code, as unit tests are cheap to write, and assist with design and structuring your code. Integration and functional tests are by comparison more expensive to write as a developer, but necessary in understanding when you have met all your acceptance criteria. Now the project I was referring to above did not have a test analyst on the project, and we therefore did not have any well defined acceptance criteria associated with a story. As a developer I hate this as I do not know when to stop writing tests, and therefore test everything by default. This resulted in a lower velocity.
The original post was written about three years ago, and I have worked on many more projects with different team structures since then. I have found it more productive when a tester is engaged on a project as part of the development team. They essentially work closely with a business analyst to define a story card. The business analyst would gather the requirements and flesh out the narrative for a story card, and the test analyst would sit in the same meetings with the business and define the acceptance criteria. This would generally happen in the iteration before any work is picked up by the developers. The tester becomes one of the most important roles on the project team, as they help provide focus for the development team, which streamlines the amount of code that is required to be written to meet a requirement.
In the following iteration, when a developer picks up a new card to work on, they will first discuss the details with the tester and then start work. As a developer I would write my unit tests, and with the guidance from the tester I would write just enough integration and functional tests to meet the acceptance criteria. This leaves the tester with more time to spend on exploration testing, and less time on regression tests as a majority of the test suite would have been automated by the developer. If something fails during exploration testing, then the card goes back to the developer to first write a test case to reproduce the bug, then write code to fix it. This confirms that you will never raise the same defect again.
With the approach described above you will effectively get more test cycles through out the life of the project. This was demonstrated on a recent project that ran for about 14 weeks with one week iterations. We essentially completed 14 test cycles out of the project and went through UAT once. By comparison a similar “Waterfall” project had four test cycles scheduled at the end by the test team, but was reduced to two test cycles when it came to the crunch. This project failed late, and subsequently costs and delivery dates blew out when it came to fixing the defects and getting bounced out of UAT numerous times.
If you are a tester in a test team that takes a deliverable from a development team to test, then obviously doing just enough testing won’t work, especially if the development team do not write enough tests. If you want better quality assurance baked into the software development lifecycle, then embed a tester in the project team and get your developers to do Test Driven Development. Although this is sometimes easier said than done depending on an organisation’s culture and structure.
Similar thoughts on this topic from other developers:
Automated Testing and the Test Pyramid (http://jamescrisp.org/2011/05/30/automated-testing-and-the-test-pyramid/) by James Crisp.
Testing like the TSA (http://37signals.com/svn/posts/3159-testing-like-the-tsa) by DHH.