Nick Carroll

I have been working as part of a team at ThoughtWorks to build an iPhone app for the upcoming Agile Australia 2009 conference. It has been an anxious wait for Apple to approve the app, but it was a thrill when that email finally came through. I am pleased to announce that the Agile Australia 2009 iphone app is now available for download at the iTunes store for free! If you are attending the conference then this companion app will keep you informed of the schedule, provide feedback, follow the hot sessions, find your rooms, and tweet about the talks.

Hoorah! Just got my new work laptop. It is a brand spanking new 15 inch Macbook pro! Apparently it is the new default laptop for developers at ThoughtWorks Australia.

This guide is basically a rehash of this posting in Joyent’s support forums. I am reproducing it below to include my experiences as I found some discrepancies with the posting in the Joyent forums.

Joyent Shared Accelerators don’t allow you to deploy your Django app using mod_python, so you have to create a proxy path that diverts traffic to lighttpd and FastCGI to serve your Django app.

For this guide you will need to replace ${USER} with your username on Joyent’s server, the hostname of your server as ${HOST}, and your DNS domain as ${DOMAIN}.

Set up Apache as a proxy server for lighttpd

You no longer need to submit a support ticket to request a port number for lighttpd. You can just go to Virtualmin for your server (https://virtualmin.joyent.us/${HOST}/) > Other Tools > Check ports to view a list of available port numbers that have been reserved for you. Pick one and note it down. We will refer to this port number as ${PORT}.

Set up a directory structure for lighttpd:

mkdir -p ~/etc/init.d
mkdir -p ~/etc/lighttpd/vhosts.d
touch ~/logs/lighttpd.error.log ~/logs/lighttpd.access.log

Using a text editor, create ~/etc/lighttpd/lighttpd.conf:

#-- Lighttpd modules

server.modules = ( "mod_rewrite",
                                "mod_redirect",
                                "mod_access",
                                "mod_cgi",
                                "mod_fastcgi",
                                "mod_compress",
                                "mod_accesslog",
                                "mod_alias" )

#-- Fundamental process configs
server.port = ${PORT}
server.username = "${USER}"
server.groupname = server.username
var.base = "/users/home/" + server.username
server.pid-file = base + "/var/run/lighttpd.pid"

#-- Logging
server.errorlog = base + "/logs/lighttpd.error.log"
accesslog.filename = base + "/logs/lighttpd.access.log"

#-- Default
server.document-root = base + "/web/public"
server.indexfiles = ( "index.php", "index.html",  "index.htm", "default.htm" )

#-- Security
url.access-deny = ( "~", ".inc", ".ht" )

#-- Mimetypes
include_shell "cat " + base + "/etc/lighttpd_mimetypes.conf"

#-- VHOSTS

Create ~/etc/lighttpd/mimetypes.conf:

mimetype.assign             = (
".pdf"          =>      "application/pdf",
".sig"          =>      "application/pgp-signature",
".spl"          =>      "application/futuresplash",
".class"        =>      "application/octet-stream",
".ps"           =>      "application/postscript",
".torrent"      =>      "application/x-bittorrent",
".dvi"          =>      "application/x-dvi",
".gz"           =>      "application/x-gzip",
".pac"          =>      "application/x-ns-proxy-autoconfig",
".swf"          =>      "application/x-shockwave-flash",
".tar.gz"       =>      "application/x-tgz",
".tgz"          =>      "application/x-tgz",
".tar"          =>      "application/x-tar",
".zip"          =>      "application/zip",
".mp3"          =>      "audio/mpeg",
".m3u"          =>      "audio/x-mpegurl",
".wma"          =>      "audio/x-ms-wma",
".wax"          =>      "audio/x-ms-wax",
".ogg"          =>      "audio/x-wav",
".wav"          =>      "audio/x-wav",
".gif"          =>      "image/gif",
".jpg"          =>      "image/jpeg",
".jpeg"         =>      "image/jpeg",
".png"          =>      "image/png",
".xbm"          =>      "image/x-xbitmap",
".xpm"          =>      "image/x-xpixmap",
".xwd"          =>      "image/x-xwindowdump",
".css"          =>      "text/css",
".html"         =>      "text/html",
".htm"          =>      "text/html",
".js"           =>      "text/javascript",
".asc"          =>      "text/plain",
".c"            =>      "text/plain",
".conf"         =>      "text/plain",
".text"         =>      "text/plain",
".txt"          =>      "text/plain",
".dtd"          =>      "text/xml",
".xml"          =>      "text/xml",
".mpeg"         =>      "video/mpeg",
".mpg"          =>      "video/mpeg",
".mov"          =>      "video/quicktime",
".qt"           =>      "video/quicktime",
".avi"          =>      "video/x-msvideo",
".asf"          =>      "video/x-ms-asf",
".asx"          =>      "video/x-ms-asf",
".wmv"          =>      "video/x-ms-wmv",
".bz2"          =>      "application/x-bzip",
".tbz"          =>      "application/x-bzip-compressed-tar",
".tar.bz2"      =>      "application/x-bzip-compressed-tar"
)

Finally, create an init script at ~/etc/init.d/lighttpd:

#!/bin/sh

HOME=/users/home/${USER}
LIGHTTPD_CONF=$HOME/etc/lighttpd/lighttpd.conf
PIDFILE=$HOME/var/run/lighttpd.pid

case "$1" in

    start)
    # Starts the lighttpd daemon
    echo "Starting lighttpd"
    PATH=$PATH:/usr/local/bin /usr/local/sbin/lighttpd -f $LIGHTTPD_CONF

;;
    stop)
    # stops the daemon bt cat'ing the pidfile
    echo "Stopping lighttpd"
    kill `/bin/cat $PIDFILE`

;;
    restart)
    ## Stop the service regardless of whether it was
    ## running or not, start it again.
    echo "Restarting lighttpd"
    $0 stop
    $0 start

;;
    reload)
    # reloads the config file by sending HUP
    echo "Reloading config"
    kill -HUP `/bin/cat $PIDFILE`

;;
    *)
    echo "Usage: lighttpd (start|stop|restart|reload)"
    exit 1
;;
esac

Don’t forget to make the init script executable:

chmod 755 ~/etc/init.d/lighttpd

Proxy Apache to lighttpd

Open up a web browser, and log into https://virtualmin.joyent.us/${HOST}/

Select the virtual server to configure. Then go to Server Configuration > Proxy Paths > Add a new proxy path. Enter the following values and click Create.

Local URL path: /
Destination URLs: http://${DOMAIN}:${PORT}

Configure Django environment

Add the path to your Django app to the PYTHONPATH. Add the following to your .profile and .bashrc files.

export PYTHONPATH=/users/home/${USER}/src/django_projects

Deploying your Django app

Check out your django app to /users/home/${USER}/src/django_projects. I will refer to this django app as ${APPNAME}.

cd ~/src/django_projects
svn co svn+ssh://subversion_repos/site/${APPNAME}/trunk ${APPNAME}

Create a MySQL database

Normally I would use PostgreSQL cos it rocks, but unfortunately Joyent only provides database restrictions for specific users on MySQL. So we’ll create a mysql user and grant it privileges to access a mysql database.

Open up a web browser and log into https://virtualmin.joyent.us/${HOST}/

Select ${DOMAIN} from the dropdown list > click “Edit Databases” > Click “Create a new database”.

I entered “production” into the Database name field so that my database will be called ${USER}_${DOMAIN}_production. Then click “Create”.

Create a database user

In Virtualmin, Select ${DOMAIN} from the dropdown list.
Click “Edit Mail and FTP Users” > “Add a user to this server”.
Under Virtual domain user mailbox details, enter “django” into the Email address field. This will create a mysql user called django-${DOMAIN}, and the auto-generated password will also be used for the mysql password in your django settings.py.

Expand “Quota and home directory settings”. Limit the user’s home directory quota to 1MB.
Expand “Other user permissions”. Allow the user access to the “${USER}_${DOMAIN}_production” database we just created. Click create.

Configure project settings

In settings.py modify your database settings to the following:

FORCE_SCRIPT_NAME=''

import os.path
ROOT_DIR = os.path.abspath(os.path.dirname(file))

DATABASE_ENGINE = 'mysql'
DATABASE_NAME = '${USER}_${DOMAIN}_production'
DATABASE_USER = 'django-${DOMAIN}'
DATABASE_PASSWORD = 'password"
DATABASE_HOST = ''
DATABASE_PORT = ''

MEDIA_ROOT = os.path.join(ROOT_DIR, 'media')
MEDIA_URL = '/media/'
ADMIN_MEDIA_PREFIX = '/media/admin/'

TEMPLATE_DIRS = (
    os.path.join(ROOT_DIR, 'templates'),
)

INSTALLED_APPS = (
    'django.contrib.sites',
    'django.contrib.admin',
    'django.contrib.flatpages',
)

Since the settings file has our MySQL password inside, don’t let others read it:

chmod 600 ~/src/django_projects/${APPNAME}/settings.py

Then create the database tables in the usual fashion:

./manage.py syncdb

Create project init script

Create ~/src/djangoprojects/${APPNAME}/etc/init.sh:

#!/bin/sh

HOME="/users/home/${USER}" # Edit to your own username
PYTHONPATH=$HOME/src/django_projects
export PYTHONPATH

PROJECT_NAME="${APPNAME}"
PROJECT_DIR="$HOME/src/django_projects/$PROJECT_NAME"
PID_FILE="$HOME/var/run/$PROJECT_NAME.pid"
SOCKET_FILE="$HOME/tmp/$PROJECT_NAME.socket"
MANAGE_FILE="$PROJECT_DIR/manage.py"
METHOD="prefork"

case "$1" in

    start)
    # Starts the Django process
    echo "Starting Django project $PROJECT_NAME"
    python $MANAGE_FILE runfcgi maxchildren=2 maxspare=2 minspare=1 method=$METHOD socket=$SOCKET_FILE pidfile=$PID_FILE

;;
    stop)
    # stops the daemon by cat'ing the pidfile
    echo "Stopping Django project $PROJECT_NAME"
    kill `/bin/cat $PID_FILE`

;;
    restart)
    ## Stop the service regardless of whether it was
    ## running or not, start it again.
    echo "Restarting Django project $PROJECT_NAME"
    $0 stop
    $0 start

;;
    *)
    echo "Usage: init.sh (start|stop|restart)"
    exit 1

;;
esac

Make the init script executable:

chmod 755 ~/src/djangoprojects/${APPNAME}/etc/init.sh

Offload static media to lighttpd

We don’t want Django to be serving static content, so any path that refers to static content will be served by the web server directly from ~/web/public.

Create a softlink from django’s admin media to ~/web/public/media/admin.

mkdir ~/web/public/media
ln -s /usr/local/lib/python2.5/site-packages/django/contrib/admin/media/ ~/web/public/media/admin

Create a softlink from your project’s media directory to ~/web/public/media/public.

mkdir -p ~/src/django_projects/project/media/public
ln -s /users/home/${USER}/src/django_projects/${APPNAME}/media/public ~/web/public/media/public

Configure lighttpd

Edit ~/etc/lighttpd/vhosts.d/${APPNAME}.conf.

$HTTP["host"] =~ "(www.)?${DOMAIN}" {
    server.document-root = base + "/web/public"
    fastcgi.server = (
        "/${APPNAME}.fcgi" => (
            "main" => (
                "socket" => base + "/tmp/${APPNAME}.socket",
                "bin-environment" =>
                            ( "TZ" => "America/Chicago" ),
                "check-local" => "disable",
            )
        ),
    )

    url.rewrite-once = (
        "^(/media/admin.*)$" => "$1",
        "^(/media/public.*)$" => "$1",
        "^/favicon.ico$" => "/media/public/img/favicon.ico",
        "^(/.*)$" => "/${APPNAME}.fcgi$1",
    )
}

Then include vhosts.d/${APPNAME}.conf in your lighttpd.conf:

echo 'include "vhosts.d/${APPNAME}.conf"' >> ~/etc/lighttpd/lighttpd.conf

Schedule service start

Create a Joyent bootup action in Virtualmin. In Virtualmin, Select ${DOMAIN} from the dropdown list > go to Services > Booup Actions > Add a new bootup action. Enter the following values in the input fields and click “Create”.

Action name: init-${APPNAME}-django-site
Description: Init ${APPNAME} Django Site
Commands to run at startup: /users/home/${USER}/src/django_projects/${APPDNAME}/etc/init.sh start

Go back to the Bootup Actions, click “Add lighttpd”. Enter the following values in the input fields and click “Create”.

Action name: lighttpd-${APPNAME}-django-site
Description: Lighttpd ${APPNAME} Django Site
Commands to run at startup: /usr/local/sbin/lighttpd -f /users/home/${USER}/etc/lighttpd/lighttpd.conf

Open up your browser and go to your newly deployed Django app!

If you don’t see your site then you will have to do some debugging. I didn’t get it first time as you’ll note that my instructions above are slightly different from the original post here.

I wish I never discovered Django templates because I shudder every time I have to use Freemarker or Velocity for my Spring MVC applications. Fortunately there is an alternative. You can use StringTemplate to generate your views in Spring MVC quite easily. StringTemplate enforces a strict separation of concerns, which therefore minimises the amount of logic that is allowed in the view, thus forcing you to do more in your controllers. Therefore your view remains purely for presentation purposes.

First things first is that you need to download StringTemplate and Antlr, and make those libraries available on your classpath. Next, extend Spring’s InternalResourceView.

import org.springframework.web.servlet.view.InternalResourceView;
import org.springframework.core.io.Resource;
import org.antlr.stringtemplate.StringTemplateGroup;
import org.antlr.stringtemplate.StringTemplate;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.util.Map;
import java.io.PrintWriter;

public class StringTemplateView extends InternalResourceView {

    @Override
    protected void renderMergedOutputModel(Map model, HttpServletRequest request,
                HttpServletResponse response) throws Exception {

        Resource templateFile = getApplicationContext().getResource(getUrl());

        StringTemplateGroup group = new StringTemplateGroup("webpages", templateFile.getFile().getParent());
        StringTemplate template = group.getInstanceOf(getBeanName());
        template.setAttributes(model);

        PrintWriter writer = response.getWriter();
        writer.print(template);
        writer.flush();
        writer.close();
    }
}

Finally, you need to configure your Spring application context to use StringTemplate to render your views.

Now you can organise your StringTemplate files that have the “.st” suffix in /WEB-INF/templates. For example, you might want to organise your templates as follows.

WEB-INF
        + templates
                + layout
                        layout.st
                + partials
                        header.st
                        footer.st
                        feedback_form.st
                contact_us.st

The layout.st template might look like the following template.

<html>
    <body>
$partials/header()$
$body$
$partials/footer()$

    </body>
</html>

The above layout.st contains the basic structure for an HTML page on your site. You can use this one template to keep a consistent look and feel across your entire site. The layout.st template depends on the header.st and footer.st templates that exist in the partials directory.

To insert the feedback form in feedback_form.st into the $body$ placeholder attribute for your contacts page you simply create a contact_us.st template with the following line.

$layout/layout(body=partials/feedback_form())$

When your controller handles the request to display the contact us page, the controller will return a ModelAndView with the view name set to “contact_us”. The view name maps to the contact_us.st file in your templates directory. StringTemplate will render the contact_us page using the layout template with the header, footer and feedback form templates.

It makes me very happy to see others championing continuous deployment. Apparently the developers at IMVU release into production a number of times a day, as often as 50 releases in a day. Smells like a lot of bug fixing than new features, but at the very least they have a lean process for releasing into production. I believe having the capability to release into production as often as possible is more important than having a process that draws out the length of time that it takes to release into production.

I’ve always been a fan of getting code into production as quickly as possible. Releasing into production early and often adds business value and keeps users happy. Adding new functionality that makes it easier to add items to a shopping cart for example means the business can encourage more fluid sales through a better online experience. It also means certain bugs won’t provide a detrimental user experience that would drive customers away. You will drive more customers away the longer a bug stays in production. Or if you are lucky, have one of your users get so ticked off with your application that they provide their own patch and release it for you.

My preferred approach to continuous deployment is to integrate it as part of your continuous integration (CI) build box. With Cruise or Bamboo create a build that runs an ant script that creates a branch, compiles source, runs tests, and deploys on a successful build. You might want to keep this build separate from your regular trunk builds, as you may not want to release every time someone checks code into the source code repository, especially when your team gets into a healthy habit of doing atomic commits.

Utilising your CI box in your deployment process effectively means you can release into production with one click of a button. Imagine that, one click deployment, one click that automates your quality assurance process (assuming you have a quality suite of unit, integration, and functional tests), and production artifacts such as WAR files and configuration properties for archiving. Spending time improving your deployment process and making it leaner means you can realise business value earlier and keep your customers happier.

Isaiah, a friend from uni days and colleague at ThoughtWorks published a good article at MSDN magazine on using mocks and tests to design role-based objects.

Holy crap! I was just perusing the bin directory of my JDK installation and stumbled on a tool called VisualVM. It is a free lightweight profiling tool for the JVM. I can easily find which Java process maps to a specific Java application. Better yet the tool tells me what the PIDs are so that I can easily terminate “rogue” Java processes. VisualVM also provides visualisation tools for monitoring Java Classes, Threads, PermGen, and the Heap. It saves me from having to buy JProfiler or setting up Netbeans to profile my Java applications.

Last night at the Groovy Group we had Graeme Roche from SpringSource and lead developer of Grails give a webinar titled “Building Twitter with Grails in 40 minutes“. We were all impressed, not only with how Grails leverages Spring and third party plugins but also with the Twitter application that Graeme was building. He gave a new meaning to TDD — Twitter Driven Development. Each new feature that he added allowed him to tweet about it in the application that he was building. It was evident that he enjoys talking about Grails and connecting with user groups, even the flu wasn’t going to keep him away from giving the talk!

I am amazed with how many quality presenters we have had at the Sydney Groovy Group, and the number of project leads that have presented on their Groovy projects.

At the last meeting we got to hear about Griffon from James Williams, one of the project’s lead developers. Griffon is a Grails like framework for developing Swing based desktop applications. I was extremely impressed with Griffon, and how easy it was to create components without having to implement a bunch of event listeners or interfaces that you don’t need or care about. It was also interesting to see how the MVC pattern was applied to a Swing application. It made me wish Griffon was around 10 years ago when I was building Swing applications for Xylogy.

The May Sydney Groovy group will be meeting this Wednesday. Graeme Rocher will be giving a webinar on Grails, plus a run down on the recent GR8 Conference. So you’ll get to hear first hand on where Spring intends to take Groovy and Grails.

If you intend to make it to the meeting then please express your interest in the forums.

Hah, I knew someone would throw the navigator role at me. It maybe just a personal thing, but I certainly don’t buy into the navigator role at all. I’m sure someone that liked to hog the keyboard came up with that one.

My esteemed colleague Mark rebutted my post stating that pair programming is not about equal keyboard time, and went on to describe a case where it is perfectly alright to spend more time driving when there is a difference in experience in the pairs. A mighty good example, but really, how does his story about working with a more experienced developer trump my story about working with a more experienced developer?

Besides my thoughts on sharing the keyboard should not be taken too literally. I was not promoting equal keyboard time. I was promoting equal opportunity for both pairs to contribute, regardless of experience. Using a chess clock allows the person that has been “navigating” too long to say “look mate you have been driving for too long and here is the quantitative data to prove it”.

It sounds like Mark’s experienced pair told him how to solve the problem at hand, whereas my experienced pair taught me the thought process of going about solving the problem. This was done by having to implement code to pass my pair’s test, and to test my understanding of what was going on I had to write a test to keep the momentum going.

I don’t see the benefit where I have to write a test and then make the test that I wrote pass. Which is what happens when you hog the keyboard. The code that you produce becomes a self-fulfilling prophecy. Why not make your pair more useful by engaging her in your thought process too, by seeing if they will make your test pass the way you think it should pass? If there is a difference, then that is a good thing. That person thought of something you did not. Or your test simply wasn’t as good as you thought it would be.

A more experienced person can always steer you in the right direction with a good test. In my story, my experienced pair did just that, and if we were using a clock at the time it would show that I would have spent more time at the keyboard. I have since learned a few tricks, and I’m sure now the chess clock will indicate a more balanced reading.

Allowing one person to utilise the keyboard more than the other because of experience is flawed. If the person driving is the more experienced then forget about the less experienced pair navigating. When have you seen a less experienced developer try to influence the thoughts of the more experienced developer? More often than not the less experienced pair gets shut down the moment they pipe up, and the pair continues on with the vision of the more experienced developer that is driving. This is the classic surgeon model of software development, and it has been criticised for its lack of knowledge sharing.

If on the other hand both pairs have an equal opportunity of influencing the other through code, then the process of churning out code becomes a more engaging one for both people. If I were to give a name for this style of pair programming, then I would call it the Sudbury model of pair programming, named after the Sudbury model of education, which promotes a democratic structure for learning which is experiential based.

I have also been told in my last post that I should deliberately take control of the keyboard for extended periods of time, as this will make me a better pair. I’m sorry but how does this benefit either pair? Are you even aware of the average attention span of a human being? Why do you think that the Roads and Traffic Authority (RTA) advise you to rotate drivers every two hours? Or that class times are limited to 50 minutes? Or that stand ups should only last a few minutes. Or that a micro-sleep kills in seconds? The mind cannot stay focused on a particular task for long periods of time without some form of stimulus. The same goes for watching someone code for extensive periods.

I have also been told that the pair should play an “active” part by holding the vision for the work being done and doing the strategic thinking. What is the strategic thinking that you have to do during development? You should have already done the strategic thinking before you even started the story. The strategic thinking occurs when you are sizing up the story, and writing technical tasks that you think will need to happen in order to complete the story. If you are doing your strategic thinking during development then maybe your story is too big and can be broken down further.

Pair programming should be more like chess. Each person has to think a few moves ahead. The more experienced developer is capable of thinking several more moves ahead than the inexperienced developer, and when it is their turn they are able to lead the other down a path in discrete steps. When my pair has the keyboard I am thinking about the next test to write, possibly the next couple. Then when I get the keyboard I get to solve my pair’s test, then immediately move on to writing one of my tests for my pair to work on. This is how I like to work, and I shouldn’t need to rely on a chess clock to work this way. The chess clock is simply a tool for exposing the rocks and making it glaringly clear that the conversation has been one sided for too long.

Finally, hogging the keyboard is bad when you are working with someone that has never pair programmed before. They will never see the benefits of it as they will walk away from the experience feeling less productive. Or feel that they are not worthy enough to touch the keyboard. Constantly swapping the keyboard between pairs is more inclusive and allows for dialogue. Hogging the keyboard on the other hand is more of a monologue that trumpets “this is how you do it…”. Engaging the client developer and treating them as an equal goes a long way with relationship building.

[EDIT] Mark does a better job of explaining how to make a test pass quickly and why it is a good thing. My idea of using a chess clock was aimed to keep this in the minds of developers during a pair programming session.