One click deployment with Maven and Bamboo
A while back I wrote about achieving continuous deployment with one-click deployments. I didn’t provide an example for that post as I mostly wrote about why you need to achieve continuous deployment. Here I will follow up with a simple example of how you can achieve continuous deployment.
Continuous deployment is quite easy to setup if you are using a typical Maven project structure and Bamboo as your continuous integration tool. Also I am assuming that you want to deploy your application to a tomcat server.
In your pom.xml file add the following configuration so that you use the Tomcat plugin for deploying your application to http://hostname.com/app. Change the path, url, and server configurations to suit your needs.
org.codehaus.mojo tomcat-maven-plugin /app http://hostname.com/manager deployment.server
Also make sure that your .m2/settings.xml file contains the following for authenticating with the Tomcat manager.
deployment.server tomcat password
In Bamboo create a new pan for your project. I tend to give this plan the name “Promote to Production”. Configure the Builder goal for Maven to run clean tomcat:redeploy. I also only allow a specific user to trigger this plan so that not everyone has permission to deploy into production. Finally, configure the build strategy to run manually, so an authorised person can click on the Build plan button in Bamboo to deploy the application.
Once set up the above instructions will allow an authorised person in Bamboo to click on a single button to deploy into production. Leveraging your continuous integration tool for deployment allows you to archive deployment artifacts such as your WAR files in the case where you have to revert to a previous version.
Deploying Maven modules
My most recent consulting gig brought me back into contact again with Maven. I have this love-hate relationship with Maven. I love using Maven for simple projects, that generally have only a development environment, and I hate using Maven for more complex projects where you have to create profiles for multiple testing environments. Maven is just too restrictive for the latter case.
If you have your own internal Maven repository setup and you want to deploy reusable modules to it then follow these instructions. It took me a while to figure them out as there is something wrong with Wagon’s internal ssh implementation. Using an external scp tool will save you a lot of grief.
1. Setup your local .m2/settings.xml as follows:
internal.maven.repos maven location of your ssh private key
You’ll need to create an ssh key. And provide a reference to your private key using the
2. Log into your server hosting your maven repository and add the contents of your public key (id_rsa.pub) at the bottom of the authorized_keys2 file. You should create a generic maven user that everyone on your team can log into, this is because when you deploy your module it will be written to the filesystem with read/write permissions only for that user. Exit your server.
3. Go to your project and add the following to your project’s pom.
... internal.maven.repos Internal Maven Repository scpexe://maven@hostname.com/home/maven/.m2/repository org.apache.maven.wagon wagon-ssh-external 1.0-beta-2
4. Run mvn deploy.
5. You should now see your project deployed to the maven repository on your server.
Cloning a Java object
Here is a simple and effective approach to cloning objects in Java using XStream. The process uses XStream to serialize your object to XML, and then using the XML to create a new Java object that is a deep copy of the original.
Sheep dolly = new Sheep("Dolly");
XStream xstream = new XStream();
String xml = xstream.toXML(dolly);
Sheep newDolly = (Sheep)xstream.fromXML(xml);
Atlassian Starter License
I’m back at the University of Sydney doing some independent Agile and TDD coaching. The first thing that I wanted to set up was a CI server. My initial recommendation was Atlassian’s Bamboo product, which may come as a surprise given that I used to work for ThoughtWorks – the makers of Cruise – for the last three years. But I must confess, I have never actually used Cruise on a project, it was always Bamboo, Hudson, or Luntbuild. So I figured stick with what you know.
I thought the research group would be strapped for cash and considered setting up Hudson, but Atlassian’s Starter license gives you access to their products for an amazingly low price of $10 per product. It was easy to convince the group to use Bamboo for that price. Best of all the proceeds of the purchase price go towards a charity called Room to Read. Thank you Atlassian for your generosity!
Last month at ThoughtWorks
I was sifting through my blog entries during the holidays and came across my post three years ago about my first month at ThoughtWorks. After reading it I felt it needed an update, otherwise I probably wouldn’t have left the company if the list was still accurate. My last day at ThoughtWorks was 24th December 2009.
1. Other ThoughtWorkers: I have had the chance to work with some of the most incredible software developers that I have ever met. These guys and gals really know how to thrash a keyboard around when pumping out quality code. It has to be said that working with really talented individuals makes work very enjoyable. Which goes to show that Roy’s social experiment is still going strong after all these years.
The people at ThoughtWorks are certainly a great bunch of people. I would still rank working with many of the fine folks at ThoughtWorks as one of my best experiences. However, many of them have since moved on to greener pastures.
2. Ruby is such a cool programming language and I am so glad that I am working at a company that has completely embraced it. Ruby has been around in the US and Europe for a while, but it is still relatively new in Australia. I am waiting impatiently for the Aussie tech industry to catch up so that I can finally work on a Ruby project.
Meanwhile I have been ramping up on my Ruby skills, and there is no better place in Australia for learning Ruby. We have some of the best Ruby developers working in Sydney at the new ThoughtWorks Studios, and they open their doors after work each week for us to work on a Ruby project with them.
Sadly, I never got to write a line of Ruby code that made it into production. It was only ever Java code. Also, ThoughtWorks Studios is no longer based in Sydney.
3. Agile is certainly an interesting approach to developing software. I have already been thrown head first into a large agile software development project, and I am already hooked on the agile koolaid. Over the weekend our client deployed the first release in time for a public launch of their system. It is the first large scale project that I have worked on, and it was a success right off the bat. The release was completed on-time, on-budget, and with 5 times less defects than any of their previous projects. Being part of a team that produced a result like this certainly made my first month a memorable one.
I am still a strong advocate for Agile software development. I was never part of a project that failed to deliver, and I am extremely proud of this success.
4. Open Communication: I have been amazed with how open the lines of communication are within the company. In the first two weeks I had met Martin Fowler our lead scientist, coffee with Bruce the Australian MD, and beers with Roy Singham the founder of the company. They all made themselves available to talk about agile, corporate strategy, and whatever the next big thing might be. I thought it was cool to be able to hang out with these guys. There aren’t many companies out there that have corporate leaders that like to hang out with their employees and have a general interest in them.
Communication is not as open as it used to be.
5. Geek Night is just one of many ThoughtWorks events that helps to facilitate knowledge sharing within the company. It is an after work event where developers head back to the ThoughtWorks office for a night of geeking out in front of a computer. We learn a new programming language whilst downing copious amounts of Coke Zero and rice cracker snacks.
Geek Night rarely happens. I tried my best to inject some life into it with the Functional Programming and Groovy user groups, but the interest just wasn’t there.
6. Free food: I have put on a couple of kilos since I started work. It has to do with all the lunches and dinners that the company has put on. They keep a fridge stocked with beers, wine, soft drinks and fruit juice. As well as bowls of fruit and cupboards filled with biscuits, chips, and chocolates. Best of all they have a tab going for free coffee at a local cafe!
There is still free food on Fridays. The coffee tab at the local cafe got replaced with an in-house coffee machine.
7. Consulting Dojo: There is one catch to the free food, on Fridays lunch is provided to attract all the ThoughtWorkers to the office to listen to one of our colleagues present on a consulting related topic. The presentations have been very interesting, and there is always a good turn out. The dojo is another example of knowledge sharing within the company.
The dojos are still around, but in a lighter form. A couple of presenters are given 5 minutes to present on a topic that ranges from cool stuff that happened on a project to how to use a Mac.
8. Training and book budgets: Each ThoughtWorker is provided with their own personal budget for training and purchasing books. Essentially we are responsible for our own learning and personal development. So we can choose whatever course or conference we want to attend. There is just so much support for our career development.
The training and book budgets are still around. There was a period when the training budget was reduced, but it is back to where it was. I only ever got to use my training budget in my first year, after that I found it hard to find time for training as I was always on a project and focused on delivery. However I made good use of the book budget. It is something that I will certainly miss.
9. Brand new Dell Latitude D620: It isn’t a Mac, but I have been very impressed with this piece of hardware. I really like the widescreen display, Intel Core Duo processor, and the feel of the keyboard. It also doesn’t get that hot from extended usage. It is the first Dell I’ve used, and I’m very happy with it.
Would you believe that the Macbook Pro is now the default laptop? It was a sad day when I had to return my new 15 inch Macbook.
10. Mobile phone and home broadband: This is a nice perk, having your company pay for your mobile phone and home broadband bills. I can finally afford ADSL2!
You still get a mobile phone and broadband budget. So you can sign up for an iPhone when you join.
This will be my last post on ThoughtWorks as I close this chapter of my life.
Git on Joyent Shared Accelerator
This is just a quick guide to setting up a remote Git repository on a Joyent shared accelerator.
Log into your Joyent server and create your remote git repository.
ssh host.joyent.us
cd ~/git
mkdir project.git
cd project.git
git --bare init
chmod -x hooks
exit
Now on your local box create your project directory and push the source files to your newly created remote repository.
rails project
cd project
git init
git add .
git commit -a
git remote add origin ssh://host.joyent.us/home/username/git/project.git
git push origin master
Now get your collaborators to clone your repository and push their changes. Assuming you have added them as users via virtualmin.
git clone ssh://host.joyent.us/home/username/git/project.git
Estimation Deck
My Estimation Deck iPhone application just got published in the iTunes store! You can download it for free and use it for your next Agile estimation session. It saves me from carrying around a deck of cards, and I hope it saves you from that pain as well!
Agile Australia 2009 iPhone App
I have been working as part of a team at ThoughtWorks to build an iPhone app for the upcoming Agile Australia 2009 conference. It has been an anxious wait for Apple to approve the app, but it was a thrill when that email finally came through. I am pleased to announce that the Agile Australia 2009 iphone app is now available for download at the iTunes store for free! If you are attending the conference then this companion app will keep you informed of the schedule, provide feedback, follow the hot sessions, find your rooms, and tweet about the talks.
New Macbook
Hoorah! Just got my new work laptop. It is a brand spanking new 15 inch Macbook pro! Apparently it is the new default laptop for developers at ThoughtWorks Australia.
Deploying your Django app on Joyent Shared Accelerators
This guide is basically a rehash of this posting in Joyent’s support forums. I am reproducing it below to include my experiences as I found some discrepancies with the posting in the Joyent forums.
Joyent Shared Accelerators don’t allow you to deploy your Django app using mod_python, so you have to create a proxy path that diverts traffic to lighttpd and FastCGI to serve your Django app.
For this guide you will need to replace ${USER} with your username on Joyent’s server, the hostname of your server as ${HOST}, and your DNS domain as ${DOMAIN}.
Set up Apache as a proxy server for lighttpd
You no longer need to submit a support ticket to request a port number for lighttpd. You can just go to Virtualmin for your server (https://virtualmin.joyent.us/${HOST}/) > Other Tools > Check ports to view a list of available port numbers that have been reserved for you. Pick one and note it down. We will refer to this port number as ${PORT}.
Set up a directory structure for lighttpd:
mkdir -p ~/etc/init.d mkdir -p ~/etc/lighttpd/vhosts.d touch ~/logs/lighttpd.error.log ~/logs/lighttpd.access.log
Using a text editor, create ~/etc/lighttpd/lighttpd.conf:
#-- Lighttpd modules
server.modules = ( "mod_rewrite",
"mod_redirect",
"mod_access",
"mod_cgi",
"mod_fastcgi",
"mod_compress",
"mod_accesslog",
"mod_alias" )
#-- Fundamental process configs
server.port = ${PORT}
server.username = "${USER}"
server.groupname = server.username
var.base = "/users/home/" + server.username
server.pid-file = base + "/var/run/lighttpd.pid"
#-- Logging
server.errorlog = base + "/logs/lighttpd.error.log"
accesslog.filename = base + "/logs/lighttpd.access.log"
#-- Default
server.document-root = base + "/web/public"
server.indexfiles = ( "index.php", "index.html", "index.htm", "default.htm" )
#-- Security
url.access-deny = ( "~", ".inc", ".ht" )
#-- Mimetypes
include_shell "cat " + base + "/etc/lighttpd_mimetypes.conf"
#-- VHOSTS
Create ~/etc/lighttpd/mimetypes.conf:
mimetype.assign = ( ".pdf" => "application/pdf", ".sig" => "application/pgp-signature", ".spl" => "application/futuresplash", ".class" => "application/octet-stream", ".ps" => "application/postscript", ".torrent" => "application/x-bittorrent", ".dvi" => "application/x-dvi", ".gz" => "application/x-gzip", ".pac" => "application/x-ns-proxy-autoconfig", ".swf" => "application/x-shockwave-flash", ".tar.gz" => "application/x-tgz", ".tgz" => "application/x-tgz", ".tar" => "application/x-tar", ".zip" => "application/zip", ".mp3" => "audio/mpeg", ".m3u" => "audio/x-mpegurl", ".wma" => "audio/x-ms-wma", ".wax" => "audio/x-ms-wax", ".ogg" => "audio/x-wav", ".wav" => "audio/x-wav", ".gif" => "image/gif", ".jpg" => "image/jpeg", ".jpeg" => "image/jpeg", ".png" => "image/png", ".xbm" => "image/x-xbitmap", ".xpm" => "image/x-xpixmap", ".xwd" => "image/x-xwindowdump", ".css" => "text/css", ".html" => "text/html", ".htm" => "text/html", ".js" => "text/javascript", ".asc" => "text/plain", ".c" => "text/plain", ".conf" => "text/plain", ".text" => "text/plain", ".txt" => "text/plain", ".dtd" => "text/xml", ".xml" => "text/xml", ".mpeg" => "video/mpeg", ".mpg" => "video/mpeg", ".mov" => "video/quicktime", ".qt" => "video/quicktime", ".avi" => "video/x-msvideo", ".asf" => "video/x-ms-asf", ".asx" => "video/x-ms-asf", ".wmv" => "video/x-ms-wmv", ".bz2" => "application/x-bzip", ".tbz" => "application/x-bzip-compressed-tar", ".tar.bz2" => "application/x-bzip-compressed-tar" )
Finally, create an init script at ~/etc/init.d/lighttpd:
#!/bin/sh
HOME=/users/home/${USER}
LIGHTTPD_CONF=$HOME/etc/lighttpd/lighttpd.conf
PIDFILE=$HOME/var/run/lighttpd.pid
case "$1" in
start)
# Starts the lighttpd daemon
echo "Starting lighttpd"
PATH=$PATH:/usr/local/bin /usr/local/sbin/lighttpd -f $LIGHTTPD_CONF
;;
stop)
# stops the daemon bt cat'ing the pidfile
echo "Stopping lighttpd"
kill `/bin/cat $PIDFILE`
;;
restart)
## Stop the service regardless of whether it was
## running or not, start it again.
echo "Restarting lighttpd"
$0 stop
$0 start
;;
reload)
# reloads the config file by sending HUP
echo "Reloading config"
kill -HUP `/bin/cat $PIDFILE`
;;
*)
echo "Usage: lighttpd (start|stop|restart|reload)"
exit 1
;;
esac
Don’t forget to make the init script executable:
chmod 755 ~/etc/init.d/lighttpd
Proxy Apache to lighttpd
Open up a web browser, and log into https://virtualmin.joyent.us/${HOST}/
Select the virtual server to configure. Then go to Server Configuration > Proxy Paths > Add a new proxy path. Enter the following values and click Create.
Local URL path: /
Destination URLs: http://${DOMAIN}:${PORT}
Configure Django environment
Add the path to your Django app to the PYTHONPATH. Add the following to your .profile and .bashrc files.
export PYTHONPATH=/users/home/${USER}/src/django_projects
Deploying your Django app
Check out your django app to /users/home/${USER}/src/django_projects. I will refer to this django app as ${APPNAME}.
cd ~/src/django_projects
svn co svn+ssh://subversion_repos/site/${APPNAME}/trunk ${APPNAME}
Create a MySQL database
Normally I would use PostgreSQL cos it rocks, but unfortunately Joyent only provides database restrictions for specific users on MySQL. So we’ll create a mysql user and grant it privileges to access a mysql database.
Open up a web browser and log into https://virtualmin.joyent.us/${HOST}/
Select ${DOMAIN} from the dropdown list > click “Edit Databases” > Click “Create a new database”.
I entered “production” into the Database name field so that my database will be called ${USER}_${DOMAIN}_production. Then click “Create”.
Create a database user
In Virtualmin, Select ${DOMAIN} from the dropdown list.
Click “Edit Mail and FTP Users” > “Add a user to this server”.
Under Virtual domain user mailbox details, enter “django” into the Email address field. This will create a mysql user called django-${DOMAIN}, and the auto-generated password will also be used for the mysql password in your django settings.py.
Expand “Quota and home directory settings”. Limit the user’s home directory quota to 1MB.
Expand “Other user permissions”. Allow the user access to the “${USER}_${DOMAIN}_production” database we just created. Click create.
Configure project settings
In settings.py modify your database settings to the following:
FORCE_SCRIPT_NAME=''
import os.path
ROOT_DIR = os.path.abspath(os.path.dirname(file))
DATABASE_ENGINE = 'mysql'
DATABASE_NAME = '${USER}_${DOMAIN}_production'
DATABASE_USER = 'django-${DOMAIN}'
DATABASE_PASSWORD = 'password"
DATABASE_HOST = ''
DATABASE_PORT = ''
MEDIA_ROOT = os.path.join(ROOT_DIR, 'media')
MEDIA_URL = '/media/'
ADMIN_MEDIA_PREFIX = '/media/admin/'
TEMPLATE_DIRS = (
os.path.join(ROOT_DIR, 'templates'),
)
INSTALLED_APPS = (
'django.contrib.sites',
'django.contrib.admin',
'django.contrib.flatpages',
)
Since the settings file has our MySQL password inside, don’t let others read it:
chmod 600 ~/src/django_projects/${APPNAME}/settings.py
Then create the database tables in the usual fashion:
./manage.py syncdb
Create project init script
Create ~/src/djangoprojects/${APPNAME}/etc/init.sh:
#!/bin/sh
HOME="/users/home/${USER}" # Edit to your own username
PYTHONPATH=$HOME/src/django_projects
export PYTHONPATH
PROJECT_NAME="${APPNAME}"
PROJECT_DIR="$HOME/src/django_projects/$PROJECT_NAME"
PID_FILE="$HOME/var/run/$PROJECT_NAME.pid"
SOCKET_FILE="$HOME/tmp/$PROJECT_NAME.socket"
MANAGE_FILE="$PROJECT_DIR/manage.py"
METHOD="prefork"
case "$1" in
start)
# Starts the Django process
echo "Starting Django project $PROJECT_NAME"
python $MANAGE_FILE runfcgi maxchildren=2 maxspare=2 minspare=1 method=$METHOD socket=$SOCKET_FILE pidfile=$PID_FILE
;;
stop)
# stops the daemon by cat'ing the pidfile
echo "Stopping Django project $PROJECT_NAME"
kill `/bin/cat $PID_FILE`
;;
restart)
## Stop the service regardless of whether it was
## running or not, start it again.
echo "Restarting Django project $PROJECT_NAME"
$0 stop
$0 start
;;
*)
echo "Usage: init.sh (start|stop|restart)"
exit 1
;;
esac
Make the init script executable:
chmod 755 ~/src/djangoprojects/${APPNAME}/etc/init.sh
Offload static media to lighttpd
We don’t want Django to be serving static content, so any path that refers to static content will be served by the web server directly from ~/web/public.
Create a softlink from django’s admin media to ~/web/public/media/admin.
mkdir ~/web/public/media ln -s /usr/local/lib/python2.5/site-packages/django/contrib/admin/media/ ~/web/public/media/admin
Create a softlink from your project’s media directory to ~/web/public/media/public.
mkdir -p ~/src/django_projects/project/media/public
ln -s /users/home/${USER}/src/django_projects/${APPNAME}/media/public ~/web/public/media/public
Configure lighttpd
Edit ~/etc/lighttpd/vhosts.d/${APPNAME}.conf.
$HTTP["host"] =~ "(www.)?${DOMAIN}" {
server.document-root = base + "/web/public"
fastcgi.server = (
"/${APPNAME}.fcgi" => (
"main" => (
"socket" => base + "/tmp/${APPNAME}.socket",
"bin-environment" =>
( "TZ" => "America/Chicago" ),
"check-local" => "disable",
)
),
)
url.rewrite-once = (
"^(/media/admin.*)$" => "$1",
"^(/media/public.*)$" => "$1",
"^/favicon.ico$" => "/media/public/img/favicon.ico",
"^(/.*)$" => "/${APPNAME}.fcgi$1",
)
}
Then include vhosts.d/${APPNAME}.conf in your lighttpd.conf:
echo 'include "vhosts.d/${APPNAME}.conf"' >> ~/etc/lighttpd/lighttpd.conf
Schedule service start
Create a Joyent bootup action in Virtualmin. In Virtualmin, Select ${DOMAIN} from the dropdown list > go to Services > Booup Actions > Add a new bootup action. Enter the following values in the input fields and click “Create”.
Action name: init-${APPNAME}-django-site
Description: Init ${APPNAME} Django Site
Commands to run at startup: /users/home/${USER}/src/django_projects/${APPDNAME}/etc/init.sh start
Go back to the Bootup Actions, click “Add lighttpd”. Enter the following values in the input fields and click “Create”.
Action name: lighttpd-${APPNAME}-django-site
Description: Lighttpd ${APPNAME} Django Site
Commands to run at startup: /usr/local/sbin/lighttpd -f /users/home/${USER}/etc/lighttpd/lighttpd.conf
Open up your browser and go to your newly deployed Django app!
If you don’t see your site then you will have to do some debugging. I didn’t get it first time as you’ll note that my instructions above are slightly different from the original post here.
