Nick Carroll

This guide is basically a rehash of this posting in Joyent’s support forums. I am reproducing it below to include my experiences as I found some discrepancies with the posting in the Joyent forums.

Joyent Shared Accelerators don’t allow you to deploy your Django app using mod_python, so you have to create a proxy path that diverts traffic to lighttpd and FastCGI to serve your Django app.

For this guide you will need to replace ${USER} with your username on Joyent’s server, the hostname of your server as ${HOST}, and your DNS domain as ${DOMAIN}.

Set up Apache as a proxy server for lighttpd

You no longer need to submit a support ticket to request a port number for lighttpd. You can just go to Virtualmin for your server (https://virtualmin.joyent.us/${HOST}/) > Other Tools > Check ports to view a list of available port numbers that have been reserved for you. Pick one and note it down. We will refer to this port number as ${PORT}.

Set up a directory structure for lighttpd:

mkdir -p ~/etc/init.d
mkdir -p ~/etc/lighttpd/vhosts.d
touch ~/logs/lighttpd.error.log ~/logs/lighttpd.access.log

Using a text editor, create ~/etc/lighttpd/lighttpd.conf:

#-- Lighttpd modules

server.modules = ( "mod_rewrite",
                                "mod_redirect",
                                "mod_access",
                                "mod_cgi",
                                "mod_fastcgi",
                                "mod_compress",
                                "mod_accesslog",
                                "mod_alias" )

#-- Fundamental process configs
server.port = ${PORT}
server.username = "${USER}"
server.groupname = server.username
var.base = "/users/home/" + server.username
server.pid-file = base + "/var/run/lighttpd.pid"

#-- Logging
server.errorlog = base + "/logs/lighttpd.error.log"
accesslog.filename = base + "/logs/lighttpd.access.log"

#-- Default
server.document-root = base + "/web/public"
server.indexfiles = ( "index.php", "index.html",  "index.htm", "default.htm" )

#-- Security
url.access-deny = ( "~", ".inc", ".ht" )

#-- Mimetypes
include_shell "cat " + base + "/etc/lighttpd_mimetypes.conf"

#-- VHOSTS

Create ~/etc/lighttpd/mimetypes.conf:

mimetype.assign             = (
".pdf"          =>      "application/pdf",
".sig"          =>      "application/pgp-signature",
".spl"          =>      "application/futuresplash",
".class"        =>      "application/octet-stream",
".ps"           =>      "application/postscript",
".torrent"      =>      "application/x-bittorrent",
".dvi"          =>      "application/x-dvi",
".gz"           =>      "application/x-gzip",
".pac"          =>      "application/x-ns-proxy-autoconfig",
".swf"          =>      "application/x-shockwave-flash",
".tar.gz"       =>      "application/x-tgz",
".tgz"          =>      "application/x-tgz",
".tar"          =>      "application/x-tar",
".zip"          =>      "application/zip",
".mp3"          =>      "audio/mpeg",
".m3u"          =>      "audio/x-mpegurl",
".wma"          =>      "audio/x-ms-wma",
".wax"          =>      "audio/x-ms-wax",
".ogg"          =>      "audio/x-wav",
".wav"          =>      "audio/x-wav",
".gif"          =>      "image/gif",
".jpg"          =>      "image/jpeg",
".jpeg"         =>      "image/jpeg",
".png"          =>      "image/png",
".xbm"          =>      "image/x-xbitmap",
".xpm"          =>      "image/x-xpixmap",
".xwd"          =>      "image/x-xwindowdump",
".css"          =>      "text/css",
".html"         =>      "text/html",
".htm"          =>      "text/html",
".js"           =>      "text/javascript",
".asc"          =>      "text/plain",
".c"            =>      "text/plain",
".conf"         =>      "text/plain",
".text"         =>      "text/plain",
".txt"          =>      "text/plain",
".dtd"          =>      "text/xml",
".xml"          =>      "text/xml",
".mpeg"         =>      "video/mpeg",
".mpg"          =>      "video/mpeg",
".mov"          =>      "video/quicktime",
".qt"           =>      "video/quicktime",
".avi"          =>      "video/x-msvideo",
".asf"          =>      "video/x-ms-asf",
".asx"          =>      "video/x-ms-asf",
".wmv"          =>      "video/x-ms-wmv",
".bz2"          =>      "application/x-bzip",
".tbz"          =>      "application/x-bzip-compressed-tar",
".tar.bz2"      =>      "application/x-bzip-compressed-tar"
)

Finally, create an init script at ~/etc/init.d/lighttpd:

#!/bin/sh

HOME=/users/home/${USER}
LIGHTTPD_CONF=$HOME/etc/lighttpd/lighttpd.conf
PIDFILE=$HOME/var/run/lighttpd.pid

case "$1" in

    start)
    # Starts the lighttpd daemon
    echo "Starting lighttpd"
    PATH=$PATH:/usr/local/bin /usr/local/sbin/lighttpd -f $LIGHTTPD_CONF

;;
    stop)
    # stops the daemon bt cat'ing the pidfile
    echo "Stopping lighttpd"
    kill `/bin/cat $PIDFILE`

;;
    restart)
    ## Stop the service regardless of whether it was
    ## running or not, start it again.
    echo "Restarting lighttpd"
    $0 stop
    $0 start

;;
    reload)
    # reloads the config file by sending HUP
    echo "Reloading config"
    kill -HUP `/bin/cat $PIDFILE`

;;
    *)
    echo "Usage: lighttpd (start|stop|restart|reload)"
    exit 1
;;
esac

Don’t forget to make the init script executable:

chmod 755 ~/etc/init.d/lighttpd

Proxy Apache to lighttpd

Open up a web browser, and log into https://virtualmin.joyent.us/${HOST}/

Select the virtual server to configure. Then go to Server Configuration > Proxy Paths > Add a new proxy path. Enter the following values and click Create.

Local URL path: /
Destination URLs: http://${DOMAIN}:${PORT}

Configure Django environment

Add the path to your Django app to the PYTHONPATH. Add the following to your .profile and .bashrc files.

export PYTHONPATH=/users/home/${USER}/src/django_projects

Deploying your Django app

Check out your django app to /users/home/${USER}/src/django_projects. I will refer to this django app as ${APPNAME}.

cd ~/src/django_projects
svn co svn+ssh://subversion_repos/site/${APPNAME}/trunk ${APPNAME}

Create a MySQL database

Normally I would use PostgreSQL cos it rocks, but unfortunately Joyent only provides database restrictions for specific users on MySQL. So we’ll create a mysql user and grant it privileges to access a mysql database.

Open up a web browser and log into https://virtualmin.joyent.us/${HOST}/

Select ${DOMAIN} from the dropdown list > click “Edit Databases” > Click “Create a new database”.

I entered “production” into the Database name field so that my database will be called ${USER}_${DOMAIN}_production. Then click “Create”.

Create a database user

In Virtualmin, Select ${DOMAIN} from the dropdown list.
Click “Edit Mail and FTP Users” > “Add a user to this server”.
Under Virtual domain user mailbox details, enter “django” into the Email address field. This will create a mysql user called django-${DOMAIN}, and the auto-generated password will also be used for the mysql password in your django settings.py.

Expand “Quota and home directory settings”. Limit the user’s home directory quota to 1MB.
Expand “Other user permissions”. Allow the user access to the “${USER}_${DOMAIN}_production” database we just created. Click create.

Configure project settings

In settings.py modify your database settings to the following:

FORCE_SCRIPT_NAME=''

import os.path
ROOT_DIR = os.path.abspath(os.path.dirname(file))

DATABASE_ENGINE = 'mysql'
DATABASE_NAME = '${USER}_${DOMAIN}_production'
DATABASE_USER = 'django-${DOMAIN}'
DATABASE_PASSWORD = 'password"
DATABASE_HOST = ''
DATABASE_PORT = ''

MEDIA_ROOT = os.path.join(ROOT_DIR, 'media')
MEDIA_URL = '/media/'
ADMIN_MEDIA_PREFIX = '/media/admin/'

TEMPLATE_DIRS = (
    os.path.join(ROOT_DIR, 'templates'),
)

INSTALLED_APPS = (
    'django.contrib.sites',
    'django.contrib.admin',
    'django.contrib.flatpages',
)

Since the settings file has our MySQL password inside, don’t let others read it:

chmod 600 ~/src/django_projects/${APPNAME}/settings.py

Then create the database tables in the usual fashion:

./manage.py syncdb

Create project init script

Create ~/src/djangoprojects/${APPNAME}/etc/init.sh:

#!/bin/sh

HOME="/users/home/${USER}" # Edit to your own username
PYTHONPATH=$HOME/src/django_projects
export PYTHONPATH

PROJECT_NAME="${APPNAME}"
PROJECT_DIR="$HOME/src/django_projects/$PROJECT_NAME"
PID_FILE="$HOME/var/run/$PROJECT_NAME.pid"
SOCKET_FILE="$HOME/tmp/$PROJECT_NAME.socket"
MANAGE_FILE="$PROJECT_DIR/manage.py"
METHOD="prefork"

case "$1" in

    start)
    # Starts the Django process
    echo "Starting Django project $PROJECT_NAME"
    python $MANAGE_FILE runfcgi maxchildren=2 maxspare=2 minspare=1 method=$METHOD socket=$SOCKET_FILE pidfile=$PID_FILE

;;
    stop)
    # stops the daemon by cat'ing the pidfile
    echo "Stopping Django project $PROJECT_NAME"
    kill `/bin/cat $PID_FILE`

;;
    restart)
    ## Stop the service regardless of whether it was
    ## running or not, start it again.
    echo "Restarting Django project $PROJECT_NAME"
    $0 stop
    $0 start

;;
    *)
    echo "Usage: init.sh (start|stop|restart)"
    exit 1

;;
esac

Make the init script executable:

chmod 755 ~/src/djangoprojects/${APPNAME}/etc/init.sh

Offload static media to lighttpd

We don’t want Django to be serving static content, so any path that refers to static content will be served by the web server directly from ~/web/public.

Create a softlink from django’s admin media to ~/web/public/media/admin.

mkdir ~/web/public/media
ln -s /usr/local/lib/python2.5/site-packages/django/contrib/admin/media/ ~/web/public/media/admin

Create a softlink from your project’s media directory to ~/web/public/media/public.

mkdir -p ~/src/django_projects/project/media/public
ln -s /users/home/${USER}/src/django_projects/${APPNAME}/media/public ~/web/public/media/public

Configure lighttpd

Edit ~/etc/lighttpd/vhosts.d/${APPNAME}.conf.

$HTTP["host"] =~ "(www.)?${DOMAIN}" {
    server.document-root = base + "/web/public"
    fastcgi.server = (
        "/${APPNAME}.fcgi" => (
            "main" => (
                "socket" => base + "/tmp/${APPNAME}.socket",
                "bin-environment" =>
                            ( "TZ" => "America/Chicago" ),
                "check-local" => "disable",
            )
        ),
    )

    url.rewrite-once = (
        "^(/media/admin.*)$" => "$1",
        "^(/media/public.*)$" => "$1",
        "^/favicon.ico$" => "/media/public/img/favicon.ico",
        "^(/.*)$" => "/${APPNAME}.fcgi$1",
    )
}

Then include vhosts.d/${APPNAME}.conf in your lighttpd.conf:

echo 'include "vhosts.d/${APPNAME}.conf"' >> ~/etc/lighttpd/lighttpd.conf

Schedule service start

Create a Joyent bootup action in Virtualmin. In Virtualmin, Select ${DOMAIN} from the dropdown list > go to Services > Booup Actions > Add a new bootup action. Enter the following values in the input fields and click “Create”.

Action name: init-${APPNAME}-django-site
Description: Init ${APPNAME} Django Site
Commands to run at startup: /users/home/${USER}/src/django_projects/${APPDNAME}/etc/init.sh start

Go back to the Bootup Actions, click “Add lighttpd”. Enter the following values in the input fields and click “Create”.

Action name: lighttpd-${APPNAME}-django-site
Description: Lighttpd ${APPNAME} Django Site
Commands to run at startup: /usr/local/sbin/lighttpd -f /users/home/${USER}/etc/lighttpd/lighttpd.conf

Open up your browser and go to your newly deployed Django app!

If you don’t see your site then you will have to do some debugging. I didn’t get it first time as you’ll note that my instructions above are slightly different from the original post here.

I really like Django. It is not bloated like a lot of other frameworks, and it has a healthy balance between convention and configuration. As a developer I want to be able to use the tools that I want to use, and not be forced into a specific way of doing things. An example of this in Django is the comments framework that is part of django.contrib. The comments framework provides the infrastructure for attaching comments to any domain model in your Django project through content types. It also provides a few spam prevention features that you should consider leveraging, such as a security hash or a hidden honeypot field. However, the default comments form is rather bare, and the workflow for posting a comment requires too many clicks and redirects.

There are some solutions that try to improve the user experience, and do a good job of it, but require hacking the comments framework. I personally don’t like hacking the internals of any framework. Not because I’m scared to, but because I don’t want to inherit any unnecessary maintenance overheads.

Besides, I believe the comments framework does the right thing. It doesn’t try to do too much, and by that I mean the HTML that it generates for the default rendered form and corresponding responses can easily be mashed up as part of the submitting pages DOM. So with a bit of JQuery and AJAX knowhow you can post comments without refreshing or being redirected away from the submitting page.

First I use the utility methods from the comments framework to list all comments for a particular discussion content type, and to display the default comment form immediately after. Note that the comment form is wrapped in div tags with the id “comment_form”. The div will be used to override the block with responses from the server after an ajax post.

Comment

    {% load comments %}
    {% get_comment_form for discussion as form %}

    {% render_comment_form for discussion %}
    

In my template (discussion.html) in which I want to allow people to comment on something I add the following to a script block that will insert the JavaScript into the head element of the base template (base.html).

In line 2 I define a method called bindPostCommentHandler() which performs the ajax call to post a comment and to handle the response when the form submit event is triggered.

In line 3 I remove the preview button from the form as I do not want this functionality. Rich text plugins generally have a preview mode, so I don’t see the need for the server to process a preview request.

In line 4 I bind the ajax post call to the submit event of the comments form. This means when I click on the Post button to submit the form I will trigger an ajax post instead of a normal post to the server.

Line 7 serialises the data from the form input fields using the JQuery serialize() function.

Line 8 specifies the URL to post to. I use the utility method from the comments framework to retrieve this for me, so I don’t need to hard code it to a specific URL. The comment_form_target should retrieve the correct URL for the comments application that you should have configured in your urls.py file. If not then add the following URL route to your urls.py file.

    (r'^comments/', include('django.contrib.comments.urls')),

Line 10 specifies that the response will be HTML. This is necessary so that JQuery knows how to parse and handle the response.

Line 11 details the success callback, which is the function that calls when the response is successful. The callback simply replaces the form in the div wrapper with the response. A successful response could either be a “thank you” message for posting a comment, or a form displaying fields with invalid fields. Line 13 is necessary to rebind the bindPostCommentHandler() function to any new form that appears in the div wrapper. If you omit this rebinding, then you will lose ajax posts on successive submits.

Line 15 defines the callback that gets called when the server responds with an http error code. I just override the div wrapper block with an apologetic message.

Finally, line 23 binds the bindPostCommentHandler() function when the page is ready.

As you can see from the above, all you need is some JavaScript to improve the user experience for posting comments using Django’s comments framework. No framework hacking necessary. Note that if you want to further improve this solution, then you can add sliders, fade ins, and fade outs for displaying the responses from the server. I left these out for brevity. I also left out appending the submitted comment to the list of comments, as the above solution will only display the submitted comment after the user performs a GET request after the submit.

Search in Django has been a pain point until now. Haystack is a cool new modular search plugin for Django that integrates with your Model definitions. You specify which Models you would like indexed, and Haystack will auto-discover them and make the content search-able from the search page that comes with the plugin.

I spent my downtime working over the holidays on a social networking website. I had a lot of fun developing something that I would not do for my day to day job. First of all I was using a technology stack that I liked, and one that is not widely used in ThoughtWorks. I have been building my site using Django with PostgreSQL as my database. After using Sybase on my last project at work, I needed to cleanse myself from having used the crappiest database ever. It was great to be back using good ol’ PostgreSQL.

The main reason for using Django is to use GeoDjango, which makes handling location based data a lot easier. Especially if you plan on using a service like Yahoo’s Fire Eagle, which provides an API for location based social networks. GeoDjango uses PostGIS on top of PostgreSQL to store data in a location friendly format. The domain model and geographic data in GeoDjango makes it easy to create a point (latitude and longitude) on Earth and do things like determine if it is located within a mapped out region or not. This is essentially where I want to go, but I might be limited by my hosting service. Need to work out if I can install all the dependencies for GeoDjango on my shared server at Joyent.

I started with the look and feel of the website, so I have my css styles and branding sorted. I now need to integrate my templates with the user registration functionality. Then start building out the functionality for supporting community events. I will release more details about the site as I progress, so stay tuned!

I’m glad that I started playing around with Django a couple months ago. It has forced me to learn Python. I now feel more at home with Python than I ever did with Ruby. Learning Python has been especially useful with exploring Gnome’s API (as I run Ubuntu on my laptop). There are also some good editors for Python, but I quite like Gedit, and I have been messing around with writing a simple plugin that uses Exuberant CTags for navigating Python code. The Gedit plugins can easily be written in, you guessed it, Python.

There is a lot happening in the Python world, and the biggest news of late is Google’s Beta access to their new service App Engine. It is basically a service for hosting your Python applications. It currently supports any Python application that uses CGI. Essentially you can get a Django web application running on App Engine, but a bit of work will be needed to integrate with App Engine’s datastore.

Anyway I don’t want to ignite a flame war as I am a lone Pythonista in a company full of Rubyists.