Nick Carroll

There is nothing more boring than sitting and watching your Pair hack away at the keyboard during a “pair programming” session. You lose interest in the problem and drift away to LaLa land.

Sharing the keyboard means engaging both Pairs in solving the problem. If you aren’t sharing the keyboard and allowing your pair to interact with the story being developed, then you may as well be coding by yourself.

Keyboard hogging is generally symptomatic of a lack of test driven development (TDD). TDD is a good practice to get into when pair programming. It allows for one person to write a failing test, and the other to make the test pass. Then the roles are reversed so that each person takes it in turn to think of a test to write, so that the other can make it pass. This is sometimes called ping pong programming because the keyboard goes back and forth quite frequently.

How do you know if you have a keyboard hogger on your team? Well that was the question that I’ve been pondering on for a while. Some people just don’t know they are keyboard hoggers. So you need something visual and metrics based to expose the problem. Something like a chess clock.

I used to play chess competitively and to prevent people taking forever with their move you had to clock-in when it was your thinking time, and clock-out when you finally made your move. The clock added that extra bit of pressure to get a move on.

When the opposition is clocked-in you still have to remain actively engaged in the game as you have to work out a strategy to counteract a number of possible moves that your opponent could make.

Pair programming should be more like this. It certainly makes it a lot more enjoyable. I remember the best programming sessions I’ve had was with Stacy Curl, now an ex-thoughtworker and whom I believe was also a chess player. He would always look to quickly make my tests pass, even if it was to just echo the output that my tests would sometimes expect. This often forced me to think about triangulating my tests so that my pair had no choice but to do some heavy lifting and implement the code.

If you have a keyboard hogger on your team, then try and introduce a chess clock. The timed metrics should make it obvious that only one person is engaged in developing the story, and the other is just pondering about the meaning of life.

I’ve been using a JQuery tooltip plugin called QTip on my current project. It has that wow factor that blows users away. I highly recommend it.

I really like Django. It is not bloated like a lot of other frameworks, and it has a healthy balance between convention and configuration. As a developer I want to be able to use the tools that I want to use, and not be forced into a specific way of doing things. An example of this in Django is the comments framework that is part of django.contrib. The comments framework provides the infrastructure for attaching comments to any domain model in your Django project through content types. It also provides a few spam prevention features that you should consider leveraging, such as a security hash or a hidden honeypot field. However, the default comments form is rather bare, and the workflow for posting a comment requires too many clicks and redirects.

There are some solutions that try to improve the user experience, and do a good job of it, but require hacking the comments framework. I personally don’t like hacking the internals of any framework. Not because I’m scared to, but because I don’t want to inherit any unnecessary maintenance overheads.

Besides, I believe the comments framework does the right thing. It doesn’t try to do too much, and by that I mean the HTML that it generates for the default rendered form and corresponding responses can easily be mashed up as part of the submitting pages DOM. So with a bit of JQuery and AJAX knowhow you can post comments without refreshing or being redirected away from the submitting page.

First I use the utility methods from the comments framework to list all comments for a particular discussion content type, and to display the default comment form immediately after. Note that the comment form is wrapped in div tags with the id “comment_form”. The div will be used to override the block with responses from the server after an ajax post.

Comment

    {% load comments %}
    {% get_comment_form for discussion as form %}

    {% render_comment_form for discussion %}
    

In my template (discussion.html) in which I want to allow people to comment on something I add the following to a script block that will insert the JavaScript into the head element of the base template (base.html).

In line 2 I define a method called bindPostCommentHandler() which performs the ajax call to post a comment and to handle the response when the form submit event is triggered.

In line 3 I remove the preview button from the form as I do not want this functionality. Rich text plugins generally have a preview mode, so I don’t see the need for the server to process a preview request.

In line 4 I bind the ajax post call to the submit event of the comments form. This means when I click on the Post button to submit the form I will trigger an ajax post instead of a normal post to the server.

Line 7 serialises the data from the form input fields using the JQuery serialize() function.

Line 8 specifies the URL to post to. I use the utility method from the comments framework to retrieve this for me, so I don’t need to hard code it to a specific URL. The comment_form_target should retrieve the correct URL for the comments application that you should have configured in your urls.py file. If not then add the following URL route to your urls.py file.

    (r'^comments/', include('django.contrib.comments.urls')),

Line 10 specifies that the response will be HTML. This is necessary so that JQuery knows how to parse and handle the response.

Line 11 details the success callback, which is the function that calls when the response is successful. The callback simply replaces the form in the div wrapper with the response. A successful response could either be a “thank you” message for posting a comment, or a form displaying fields with invalid fields. Line 13 is necessary to rebind the bindPostCommentHandler() function to any new form that appears in the div wrapper. If you omit this rebinding, then you will lose ajax posts on successive submits.

Line 15 defines the callback that gets called when the server responds with an http error code. I just override the div wrapper block with an apologetic message.

Finally, line 23 binds the bindPostCommentHandler() function when the page is ready.

As you can see from the above, all you need is some JavaScript to improve the user experience for posting comments using Django’s comments framework. No framework hacking necessary. Note that if you want to further improve this solution, then you can add sliders, fade ins, and fade outs for displaying the responses from the server. I left these out for brevity. I also left out appending the submitted comment to the list of comments, as the above solution will only display the submitted comment after the user performs a GET request after the submit.

Search in Django has been a pain point until now. Haystack is a cool new modular search plugin for Django that integrates with your Model definitions. You specify which Models you would like indexed, and Haystack will auto-discover them and make the content search-able from the search page that comes with the plugin.

By default if you create a Grails application called funkysite (i.e. you did grails create-app funkysite on the command line), then when you run your application the root context is set to /funkysite. This means that your controllers such as a UserController with a show action will be available at http://localhost:8080/funkysite/user/show. Having “funkysite” as the root context is redundant, especially if you want to host your application at http://funkysite.com/funkysite/user/show.

My preference is to just have “/” as the root application context. Fortunately, you can easily configure the root application context in Grails. All you need to do is edit conf/Config.groovy and add grails.app.context = “/”. This will set your root application context to “/”, so that the above UserController and show action will be available at http://localhost:8080/user/show.

I stumbled across Paver, a build and deploy tool for Python applications. It will be presented at the upcoming Pycon. Looks the goods to me.

I have recently had the need to restrict a test to run within a certain time limit. If the test does not complete within a specific time constraint then the test should fail. The approach that I took was to run the function under test in its own thread and use a CountDownLatch to specify the duration in which the process should complete within. The code for this is as follows.

public class TimeConstrainedIntegrationTest {

    private static final int RUN_ONCE = 1;

    @Test
    public void shouldVerifyGoogleIsAlive() throws Exception {
        CountDownLatch doneSignal = new CountDownLatch(RUN_ONCE);

        new Thread(new VerifyGoogleIsAliveWorker(doneSignal)).start();

        boolean finishedInTime = doneSignal.await(10, TimeUnit.SECONDS);
        assertTrue(finishedInTime);
    }

    private class VerifyGoogleIsAliveWorker implements Runnable {
        private final CountDownLatch doneSignal;

        VerifyGoogleIsAliveWorker(CountDownLatch doneSignal) {
            this.doneSignal = doneSignal;
        }

        public void run() {
            try {
                try {
                    verifyGoogleIsAlive();
                } catch (IOException e) {
                    throw new InterruptedException();
                }

                doneSignal.countDown();
            } catch (InterruptedException ex) {
            }
        }

        private void verifyGoogleIsAlive() throws IOException {
            WebClient webClient = new WebClient();
            Page page = webClient.getPage("http://google.com");
            assertTrue(StringUtils.contains(page.getTitleText(), "Google"));
        }
    }
}

In the code above I assert that Google is alive by checking the title text for http://google.com is in fact Google. Essentially I am asserting expected behaviour, which is what you would typically do. I have encapsulated this within the verifyGoogleIsAlive() method above.

To make sure that the test runs within a specific time, the verifyGoogleIsAlive() method is called within the run method of a class that implements Runnable. In this case I have called the class VerifyGoogleIsAliveWorker. The test case shouldVerifyGoogleIsAlive() should be quite self-explanatory. First a CountDownLatch is instantiated. Then a new VerifyGoogleIsAliveWorker thread is started. The CountDownLatch then awaits for the VerifyGoogleIsAliveWorker thread to run its course and complete.

Of interest though is that the awaits() method on the CountDownLatch allows you to specify the length of time in which the CountDownLatch should wait before ending the thread. If the process doesn’t complete within the specified time, then the awaits() method will return false, and therefore fail the assertion that the test will run within a specific time. If on the other hand the process completes within the allocated time frame then the test will pass.

The above concurrency pattern should be useful in making sure that your tests run within a specific time frame.

Python code completion for Emacs. Works like a charm.

The quickest way to get code into production is not to write tests. [Waits for reader to stop shouting "That is blasphemy!".] Now hear me out, this statement is true based on the assumption that the code will run as expected. However, you cannot know with certainty that your code will run as expected without testing it. Therefore you can write code and deploy it into production quickly by not writing tests, but it does not mean that the code will be defect free. As a result you run the risk of spending an indefinite amount of time fixing defects post release. This takes your time away from developing new features that add more business value.

The opposite is to spend more time upfront writing tests. This results in a longer period of development effort to get code into production, but you have more confidence that your code will run as expected. Which leaves you more time post release to work on new functionality that will add business value, instead of being swamped with fixing defects.

Now you have two contrasting views. The first is that you can quickly deploy code into production by not writing tests, but you have almost no confidence that the code will work as expected. The second is to write tests to improve confidence, but it will take longer to realise any business value. These views provide a sliding scale for software quality, at one end you have no tests and thus almost no quality assurance. At the other end you have unit tests, integration tests, functional tests, and more tests, all of which provide you with a high degree of quality. In a perfect world you would opt for the latter. But we don’t live in a perfect world. Everything costs money. Clients have a finite amount of money to spend on projects that deliver business value, and they want the best bang for their buck.

In the Agile world business value is proportional to the development team’s velocity. The higher the velocity the more business value you’ll eventually get. On a recent project we tested everything, and I mean everything. We had unit tests, integration tests, functional tests, and end-to-end tests. We did TDD, so most of these tests were written before a single line of code was written. Our test coverage was in the 90 percentile range. We had a high degree of confidence that what went into production was bullet proof. This is not to say that there were no defects, there were some defects, but they were all resolved in a timely manner. This kind of leads me to the main point of this post, you can test the hell out of something but it doesn’t guarantee that it will lead to defect free software. In other words you can have 100% test coverage but your software may not deliver what the customer wants. Or you simply didn’t cover scenarios in which your software will be used.

On this particular project the iteration manager (Agile project manager) would always state “we should be going faster”. He was right. We had a team of highly capable developers, and some of the things we were doing was not rocket science, but the business domain was quite complex. We were not experts in the business domain, and I believe we compensated by writing a lot more tests than we needed to. Through the power of hindsight I can say this, but on the project the team was geared towards writing tests. In our retrospectives the question of “how can we go faster?” was always raised. There were many useful suggestions, but nobody dared to utter the “T” word. I was always thinking it but didn’t have the balls to say it. Is this a symptom of group think? That as a team or organisation for that matter, we believe that writing tests are good, and so writing a lot more tests must therefore be better. It hardly seems Lean.

The signs that our tests were getting in the way are now glaringly clear. We had a nightly build that ran a large suite of acceptance tests and with a large set of data. Most mornings were spent figuring out why the nightly build failed. Sometimes it was due to an integration server going down, sometimes it was due to a genuine bug being exposed by testing with a large data set. The problem here is that there were a number of false positives that took time to verify and as a result took time away from working on a story.

Another sign was that our pre-commit builds took about 15 minutes. These builds included a decent size set of Selenium based functional tests. Now 15 minutes is a hell of a lot of time to be waiting around for your build to finish. To get an idea of what it was like take this xkcd comic and replace the word “compiling” with “building”. It not only holds you up, but the rest of the team as well. We had this practice of sequential check-ins, in that you have to be holding the “check-in chicken” before you were allowed to check in any code. This queued up check-ins in that you had to wait until the chicken holder had finished their build and checked in, before you were allowed to build and check in your code. It sucked even more when there were two or three other pairs in front of you. Most of the time you would just keep working, but this prevented you from doing small atomic commits, which resulted in larger commits and conflict problems.

Tests do slow you down, and it is pure ignorance to believe otherwise. It wasn’t until my most recent project where my first instinct was to test the hell out of something. The team lead on this project asked “why do you want to write that acceptance tests?”. My initial guarded response was “are you kidding me?”. This led to a massive debate over the value of writing tests. This debate was mostly between the team lead and another developer. I was too absorbed in the concept of not having to write tests for everything. Picture a cloudy sky opening up and a ray of sunshine falling down upon me. That was how I felt when I found another person that challenged the status quo of test the mofo out of everything. The default stance that we take is test everything. Usually this is done with little thought as to whether extensive testing is really required or not. Whether the client asks for it or not they are going to get a thoroughly tested system.

This leads me to the main purpose of this post, to challenge this default stance of test everything thoroughly, no matter how long it takes. It came about through the debate that we had about testing, which led to the team lead putting us in the client’s shoes. If hypothetically you had $100,000 and needed to have your website built, would you be happy that you got one piece of functionality and about five lots of tests for that functionality? Probably not. You might be happier if you instead received three pieces of functionality and a test for each functionality. I am over trivialising this, but you should get the point. Economist would call this disposition of utility gained from having extra tests in exchange for functionality marginal utility.

As in economics I believe the law of diminishing marginal utility also applies to writing tests. In other words there comes a point where the utility of writing tests detracts from the overall goal of delivering business value. This leads me to the JET acronym that I coined which stands for Just Enough Testing. It is loosely based on the concept of JeOS or Just enough Operating System. JeOS allows you to start with a stripped down operating system to run in virtual machines, then add additional services to the OS to suit the environment in which it will be used. I would like to use JeOS as an anology as to how JET should be used. JET is a stripped down testing strategy that you must adopt, then build on this strategy to suit the project that you are on. Simply adopting the stance of test everything is not a strategy that should be employed on every project. The clients’ demands will be different in all cases, and their marginal utility for business value over tests will vary.

My JET approach is to write unit tests for designing the interactions between the class under test and its collaborators. Integration tests mostly for end-to-end testing, but with the awareness of not going nuts here. Finally having a minimal set of functional tests using Selenium to cover typical workflows through a web application for example.

Adjusting JET to your project is easy. If you have the luxury of a QA for example, then get them to spend more time in writing functional tests so that all regression tests become fully automated. If your site is quite simple on the front end then your minimal set of functional tests should give you enough confidence that your application will work as expected. If however your back end is more complex, then spend a bit more time writing integration tests in that area. JET is not about writing less tests. It is about realising that projects have constraints and you need to work effectively within those constraints, and that means thinking about whether you really need to have that functional test or not, because maybe just having an integration test will give you enough confidence as to how your code will work.

In all honesty JET is just a cool acronym that relates quite nicely to velocity. Whether JET takes off (I couldn’t help myself with that pun) or not is completely left to you to decide. I’m not the only one that thinks it is perfectly ok to compromise when writing tests. Kent Beck also thinks so in his article Where, Oh Where To Test. Kent Beck comes to the conclusion in that article that a pragmatic programmer should decide what to test and how much testing is required based on three criteria: cost; reliability; and stability. From the article: “The testing strategy that delivers the needed confidence at the least cost should win.” In other words you should do just enough testing.

I know there are plenty of test zealots out there. I’ve worked with some of them, and know their counter arguments quite intimately, and they are only sound arguments in a perfect world where a client has an infinite budget for a project. For those people I have provided the pretty pictures of jet aircraft above. For the rest I hope you find the post to be an interesting read.

I spent my downtime working over the holidays on a social networking website. I had a lot of fun developing something that I would not do for my day to day job. First of all I was using a technology stack that I liked, and one that is not widely used in ThoughtWorks. I have been building my site using Django with PostgreSQL as my database. After using Sybase on my last project at work, I needed to cleanse myself from having used the crappiest database ever. It was great to be back using good ol’ PostgreSQL.

The main reason for using Django is to use GeoDjango, which makes handling location based data a lot easier. Especially if you plan on using a service like Yahoo’s Fire Eagle, which provides an API for location based social networks. GeoDjango uses PostGIS on top of PostgreSQL to store data in a location friendly format. The domain model and geographic data in GeoDjango makes it easy to create a point (latitude and longitude) on Earth and do things like determine if it is located within a mapped out region or not. This is essentially where I want to go, but I might be limited by my hosting service. Need to work out if I can install all the dependencies for GeoDjango on my shared server at Joyent.

I started with the look and feel of the website, so I have my css styles and branding sorted. I now need to integrate my templates with the user registration functionality. Then start building out the functionality for supporting community events. I will release more details about the site as I progress, so stay tuned!