Nick Carroll

This is just a quick guide to setting up a remote Git repository on a Joyent shared accelerator.

Log into your Joyent server and create your remote git repository.


ssh host.joyent.us
cd ~/git
mkdir project.git
cd project.git
git --bare init
chmod -x hooks
exit


Now on your local box create your project directory and push the source files to your newly created remote repository.


rails project
cd project
git init
git add .
git commit -a
git remote add origin ssh://host.joyent.us/home/username/git/project.git
git push origin master


Now get your collaborators to clone your repository and push their changes. Assuming you have added them as users via virtualmin.


git clone ssh://host.joyent.us/home/username/git/project.git


This guide is basically a rehash of this posting in Joyent’s support forums. I am reproducing it below to include my experiences as I found some discrepancies with the posting in the Joyent forums.

Joyent Shared Accelerators don’t allow you to deploy your Django app using mod_python, so you have to create a proxy path that diverts traffic to lighttpd and FastCGI to serve your Django app.

For this guide you will need to replace ${USER} with your username on Joyent’s server, the hostname of your server as ${HOST}, and your DNS domain as ${DOMAIN}.

Set up Apache as a proxy server for lighttpd

You no longer need to submit a support ticket to request a port number for lighttpd. You can just go to Virtualmin for your server (https://virtualmin.joyent.us/${HOST}/) > Other Tools > Check ports to view a list of available port numbers that have been reserved for you. Pick one and note it down. We will refer to this port number as ${PORT}.

Set up a directory structure for lighttpd:

mkdir -p ~/etc/init.d
mkdir -p ~/etc/lighttpd/vhosts.d
touch ~/logs/lighttpd.error.log ~/logs/lighttpd.access.log

Using a text editor, create ~/etc/lighttpd/lighttpd.conf:

#-- Lighttpd modules

server.modules = ( "mod_rewrite",
                                "mod_redirect",
                                "mod_access",
                                "mod_cgi",
                                "mod_fastcgi",
                                "mod_compress",
                                "mod_accesslog",
                                "mod_alias" )

#-- Fundamental process configs
server.port = ${PORT}
server.username = "${USER}"
server.groupname = server.username
var.base = "/users/home/" + server.username
server.pid-file = base + "/var/run/lighttpd.pid"

#-- Logging
server.errorlog = base + "/logs/lighttpd.error.log"
accesslog.filename = base + "/logs/lighttpd.access.log"

#-- Default
server.document-root = base + "/web/public"
server.indexfiles = ( "index.php", "index.html",  "index.htm", "default.htm" )

#-- Security
url.access-deny = ( "~", ".inc", ".ht" )

#-- Mimetypes
include_shell "cat " + base + "/etc/lighttpd_mimetypes.conf"

#-- VHOSTS

Create ~/etc/lighttpd/mimetypes.conf:

mimetype.assign             = (
".pdf"          =>      "application/pdf",
".sig"          =>      "application/pgp-signature",
".spl"          =>      "application/futuresplash",
".class"        =>      "application/octet-stream",
".ps"           =>      "application/postscript",
".torrent"      =>      "application/x-bittorrent",
".dvi"          =>      "application/x-dvi",
".gz"           =>      "application/x-gzip",
".pac"          =>      "application/x-ns-proxy-autoconfig",
".swf"          =>      "application/x-shockwave-flash",
".tar.gz"       =>      "application/x-tgz",
".tgz"          =>      "application/x-tgz",
".tar"          =>      "application/x-tar",
".zip"          =>      "application/zip",
".mp3"          =>      "audio/mpeg",
".m3u"          =>      "audio/x-mpegurl",
".wma"          =>      "audio/x-ms-wma",
".wax"          =>      "audio/x-ms-wax",
".ogg"          =>      "audio/x-wav",
".wav"          =>      "audio/x-wav",
".gif"          =>      "image/gif",
".jpg"          =>      "image/jpeg",
".jpeg"         =>      "image/jpeg",
".png"          =>      "image/png",
".xbm"          =>      "image/x-xbitmap",
".xpm"          =>      "image/x-xpixmap",
".xwd"          =>      "image/x-xwindowdump",
".css"          =>      "text/css",
".html"         =>      "text/html",
".htm"          =>      "text/html",
".js"           =>      "text/javascript",
".asc"          =>      "text/plain",
".c"            =>      "text/plain",
".conf"         =>      "text/plain",
".text"         =>      "text/plain",
".txt"          =>      "text/plain",
".dtd"          =>      "text/xml",
".xml"          =>      "text/xml",
".mpeg"         =>      "video/mpeg",
".mpg"          =>      "video/mpeg",
".mov"          =>      "video/quicktime",
".qt"           =>      "video/quicktime",
".avi"          =>      "video/x-msvideo",
".asf"          =>      "video/x-ms-asf",
".asx"          =>      "video/x-ms-asf",
".wmv"          =>      "video/x-ms-wmv",
".bz2"          =>      "application/x-bzip",
".tbz"          =>      "application/x-bzip-compressed-tar",
".tar.bz2"      =>      "application/x-bzip-compressed-tar"
)

Finally, create an init script at ~/etc/init.d/lighttpd:

#!/bin/sh

HOME=/users/home/${USER}
LIGHTTPD_CONF=$HOME/etc/lighttpd/lighttpd.conf
PIDFILE=$HOME/var/run/lighttpd.pid

case "$1" in

    start)
    # Starts the lighttpd daemon
    echo "Starting lighttpd"
    PATH=$PATH:/usr/local/bin /usr/local/sbin/lighttpd -f $LIGHTTPD_CONF

;;
    stop)
    # stops the daemon bt cat'ing the pidfile
    echo "Stopping lighttpd"
    kill `/bin/cat $PIDFILE`

;;
    restart)
    ## Stop the service regardless of whether it was
    ## running or not, start it again.
    echo "Restarting lighttpd"
    $0 stop
    $0 start

;;
    reload)
    # reloads the config file by sending HUP
    echo "Reloading config"
    kill -HUP `/bin/cat $PIDFILE`

;;
    *)
    echo "Usage: lighttpd (start|stop|restart|reload)"
    exit 1
;;
esac

Don’t forget to make the init script executable:

chmod 755 ~/etc/init.d/lighttpd

Proxy Apache to lighttpd

Open up a web browser, and log into https://virtualmin.joyent.us/${HOST}/

Select the virtual server to configure. Then go to Server Configuration > Proxy Paths > Add a new proxy path. Enter the following values and click Create.

Local URL path: /
Destination URLs: http://${DOMAIN}:${PORT}

Configure Django environment

Add the path to your Django app to the PYTHONPATH. Add the following to your .profile and .bashrc files.

export PYTHONPATH=/users/home/${USER}/src/django_projects

Deploying your Django app

Check out your django app to /users/home/${USER}/src/django_projects. I will refer to this django app as ${APPNAME}.

cd ~/src/django_projects
svn co svn+ssh://subversion_repos/site/${APPNAME}/trunk ${APPNAME}

Create a MySQL database

Normally I would use PostgreSQL cos it rocks, but unfortunately Joyent only provides database restrictions for specific users on MySQL. So we’ll create a mysql user and grant it privileges to access a mysql database.

Open up a web browser and log into https://virtualmin.joyent.us/${HOST}/

Select ${DOMAIN} from the dropdown list > click “Edit Databases” > Click “Create a new database”.

I entered “production” into the Database name field so that my database will be called ${USER}_${DOMAIN}_production. Then click “Create”.

Create a database user

In Virtualmin, Select ${DOMAIN} from the dropdown list.
Click “Edit Mail and FTP Users” > “Add a user to this server”.
Under Virtual domain user mailbox details, enter “django” into the Email address field. This will create a mysql user called django-${DOMAIN}, and the auto-generated password will also be used for the mysql password in your django settings.py.

Expand “Quota and home directory settings”. Limit the user’s home directory quota to 1MB.
Expand “Other user permissions”. Allow the user access to the “${USER}_${DOMAIN}_production” database we just created. Click create.

Configure project settings

In settings.py modify your database settings to the following:

FORCE_SCRIPT_NAME=''

import os.path
ROOT_DIR = os.path.abspath(os.path.dirname(file))

DATABASE_ENGINE = 'mysql'
DATABASE_NAME = '${USER}_${DOMAIN}_production'
DATABASE_USER = 'django-${DOMAIN}'
DATABASE_PASSWORD = 'password"
DATABASE_HOST = ''
DATABASE_PORT = ''

MEDIA_ROOT = os.path.join(ROOT_DIR, 'media')
MEDIA_URL = '/media/'
ADMIN_MEDIA_PREFIX = '/media/admin/'

TEMPLATE_DIRS = (
    os.path.join(ROOT_DIR, 'templates'),
)

INSTALLED_APPS = (
    'django.contrib.sites',
    'django.contrib.admin',
    'django.contrib.flatpages',
)

Since the settings file has our MySQL password inside, don’t let others read it:

chmod 600 ~/src/django_projects/${APPNAME}/settings.py

Then create the database tables in the usual fashion:

./manage.py syncdb

Create project init script

Create ~/src/djangoprojects/${APPNAME}/etc/init.sh:

#!/bin/sh

HOME="/users/home/${USER}" # Edit to your own username
PYTHONPATH=$HOME/src/django_projects
export PYTHONPATH

PROJECT_NAME="${APPNAME}"
PROJECT_DIR="$HOME/src/django_projects/$PROJECT_NAME"
PID_FILE="$HOME/var/run/$PROJECT_NAME.pid"
SOCKET_FILE="$HOME/tmp/$PROJECT_NAME.socket"
MANAGE_FILE="$PROJECT_DIR/manage.py"
METHOD="prefork"

case "$1" in

    start)
    # Starts the Django process
    echo "Starting Django project $PROJECT_NAME"
    python $MANAGE_FILE runfcgi maxchildren=2 maxspare=2 minspare=1 method=$METHOD socket=$SOCKET_FILE pidfile=$PID_FILE

;;
    stop)
    # stops the daemon by cat'ing the pidfile
    echo "Stopping Django project $PROJECT_NAME"
    kill `/bin/cat $PID_FILE`

;;
    restart)
    ## Stop the service regardless of whether it was
    ## running or not, start it again.
    echo "Restarting Django project $PROJECT_NAME"
    $0 stop
    $0 start

;;
    *)
    echo "Usage: init.sh (start|stop|restart)"
    exit 1

;;
esac

Make the init script executable:

chmod 755 ~/src/djangoprojects/${APPNAME}/etc/init.sh

Offload static media to lighttpd

We don’t want Django to be serving static content, so any path that refers to static content will be served by the web server directly from ~/web/public.

Create a softlink from django’s admin media to ~/web/public/media/admin.

mkdir ~/web/public/media
ln -s /usr/local/lib/python2.5/site-packages/django/contrib/admin/media/ ~/web/public/media/admin

Create a softlink from your project’s media directory to ~/web/public/media/public.

mkdir -p ~/src/django_projects/project/media/public
ln -s /users/home/${USER}/src/django_projects/${APPNAME}/media/public ~/web/public/media/public

Configure lighttpd

Edit ~/etc/lighttpd/vhosts.d/${APPNAME}.conf.

$HTTP["host"] =~ "(www.)?${DOMAIN}" {
    server.document-root = base + "/web/public"
    fastcgi.server = (
        "/${APPNAME}.fcgi" => (
            "main" => (
                "socket" => base + "/tmp/${APPNAME}.socket",
                "bin-environment" =>
                            ( "TZ" => "America/Chicago" ),
                "check-local" => "disable",
            )
        ),
    )

    url.rewrite-once = (
        "^(/media/admin.*)$" => "$1",
        "^(/media/public.*)$" => "$1",
        "^/favicon.ico$" => "/media/public/img/favicon.ico",
        "^(/.*)$" => "/${APPNAME}.fcgi$1",
    )
}

Then include vhosts.d/${APPNAME}.conf in your lighttpd.conf:

echo 'include "vhosts.d/${APPNAME}.conf"' >> ~/etc/lighttpd/lighttpd.conf

Schedule service start

Create a Joyent bootup action in Virtualmin. In Virtualmin, Select ${DOMAIN} from the dropdown list > go to Services > Booup Actions > Add a new bootup action. Enter the following values in the input fields and click “Create”.

Action name: init-${APPNAME}-django-site
Description: Init ${APPNAME} Django Site
Commands to run at startup: /users/home/${USER}/src/django_projects/${APPDNAME}/etc/init.sh start

Go back to the Bootup Actions, click “Add lighttpd”. Enter the following values in the input fields and click “Create”.

Action name: lighttpd-${APPNAME}-django-site
Description: Lighttpd ${APPNAME} Django Site
Commands to run at startup: /usr/local/sbin/lighttpd -f /users/home/${USER}/etc/lighttpd/lighttpd.conf

Open up your browser and go to your newly deployed Django app!

If you don’t see your site then you will have to do some debugging. I didn’t get it first time as you’ll note that my instructions above are slightly different from the original post here.

I just completed migrating this site across to a Joyent accelerator. It was a smooth and seamless migration. All I had to do was check out the site from my subversion repository on the new server, dump the database on the old server and import the data into the database on the new server. Then update the DNS server entries on my domain name registry website. Enable PowerDNS on the new server, then disable PowerDNS on the old server. Wait a couple of minutes for the change to propagate and refresh the browser to see the site running on the new server. I verified this by adding a simple HTML page on the new server to view in my browser. All in all it took a couple of hours.

Apparently yesterday was the last day to purchase a Mixed Grill or Three Martini Lunch hosting special from TextDrive. If you missed out, then you really missed out on two excellent deals.

You snooze, you lose.

I really wish that I found this posting in the Joyent Forums earlier. It tells you how to set up email and web domain aliasing to your Joyent account.

I purchased a Mixed Grill bundle from TextDrive which included hosting, online storage for secure backups, and Joyent collaboration applications. To access your joyent applications you have to go to http://username.joyent.net, and your resulting email address would be username@username.joyent.net, which to me doesn’t look that great.

So I came across the above forum posting which shows you how to set up an alias for username.joyent.net, so that you can access your joyent applications from home.yourdomain.tld, and your emails will be username@yourdomain.tld.

If your hosting is with TextDrive, then all you need to do is submit a ticket requesting that they add an MX entry for smtp.joyent.net, and a CNAME entry for home.yourdomain.com that points to username.joyent.net.

You then have to go to your Joyent Customer Control Panel and click on the Domains tab. This allows you to add a web domain and an email domain. Your web domain will be home.yourdomain.com, and your email domain is yourdomain.com. You should also select the “this is the primary domain” option.

Finally, make sure that the email aliases set up for yourdomain.com in your TextDrive webmin match the usernames of the users in your Joyent account.

All this means I can finally make use of my massive Joyent quota, which is about 50 times bigger than my Gmail account!

The other advantage of using the Joyent account is that it has IMAP, which Gmail doesn’t have. So I can access my email not only from an ajax web application, but also from a desktop mail client. The settings for your IMAP client are:

imap server: imap.joyent.net
smtp server: smtp.joyent.net
username: yourname@yourdomain.joyent.net

The Mixed Grill specials at TextDrive apparently end on 15 January 2007. If you are looking for great hosting, service, generous quotas, and a friendly community, then I highly recommend TextDrive.

Today I migrated my site over to TextDrive. It was a simple migration, but I still need to fix up email. I guess I will just have to wait until the MX rules propagate across the network.

Anyway I should be thanking David Peterson for the hosting he has provided me for the past year. I’ve decided to have a go at podcasting, and I couldn’t chew up bandwidth that way on David’s box, which hosts a few critical sites. So I’ve moved to TextDrive to use up their bandwidth.

I came across this forum thread on how to add a subversion repository to your StrongSpace account. This means you are able to use StrongSpace as your subversion repository, instead of having to use up precious disk space in TextDrive. The following is a rehash of what was covered in that forum thread.

Start by creating a subversion repository locally.

$ svnadmin create ./repos

Then secure copy the repository to your StrongSpace account.

$ scp -r ./repos ss_user@ss_user.strongspace.com:/home/ss_user/repos

From your local computer, you can access your subversion repository on StrongSpace using the svn+ssh URL schema. For example:

$ svn list svn+ssh://ss_user@ss_user.strongspace.com/home/ss_user/repos

This little tip allows you to store your public ssh key on StrongSpace, which means you can use sftp and scp to access StrongSpace without having to type in your password each time.

From the command line, enter the following:

Generate your public and private key. You can accept the default settings, or enter a passphrase to secure your private key.

$ ssh-keygen -t dsa
$ chmod 400 ~/.ssh/id_dsa

SFTP to StrongSpace to create a .ssh directory.

$ sftp ss_user@ss_user.strongspace.com
$ mkdir .ssh
$ exit

Secure copy your public key over to StrongSpace.

$ cat .ssh/id_dsa.pub >> authorized_keys
$ scp ~/authorized_keys ss_user@ss_user.strongspace.com:.ssh/authorized_keys

We now need to CHMOD the authorized_keys file on StrongSpace so that only the user has read/write privileges. Note that when we SFTP in the following, we should no longer need to type our password in.

$ sftp ss_user@ss_user.strongspace.com
$ cd .ssh
$ chmod 600 authorized_keys
$ exit

If you want to add more keys to the authorized_keys file on StrongSpace, you’ll need to first copy it to your local computer, then concatenate the new public key to the file and copy it back to StrongSpace.

$ scp ss_user@ss_user.strongspace.com:.ssh/authorized_keys ~/
$ cat .ssh/id_dsa.pub >> authorized_keys
$ scp ~/authorized_keys ss_user@ss_user.strongspace.com:.ssh/authorized_keys