Using Docker with Upstart

My Ubuntu PC has a lot of stuff on it which makes upgrading hardware or software a pain. I've been playing with docker as a potential way to containerise the services I need such as my local development server which runs many client websites for maintenance and development; a cups server configure to print to my wireless attached printer; and some other simple services that are suitable for running in a container.

My goal in doing this is to have a set of docker images and data that can easily be moved from machine to machine. I'll be able to run my development server environment on my PC, a cloud server, or laptop or share it with others by simply copying the package over and running docker-compose up. Upgrading, or changing my PC should be a much simpler task too.

I have now achieved what I wanted and it works very well.

On my main PC, I wanted to have the sevrices start automatically, so I'm using upstart to do this, but it was a bit of a challenge to get upstart working well with docker.

Upstart can handle starting and monitoring processes that fork once or twice (expect fork, and expect daemon) but my docker processes forked many more times, so upstart thinks they have stopped almost as soon as they are started.

After a bit of searching, I eventually found the answer from johnjamesmiller here on stackoverflow

The trick is to start the container in a pre-start script, then use the main start script to simply run until the docker container stops. Upstart will monitor the PID of the main start script. A post-stop script can be used to stop the container.

Another issue I had to allow for was that my disk drive containing the docker context was not always completely mounted when the start script ran, so I added a wait loop to check it is avaliable before starting.

Here is an example of an upstart conf file using this technique:

# cups-server - docker container based local cups-server
description "my docker cups-server"
author "Ian Holden"

start on filesystem and started docker
stop on runlevel [!2345]

# Automatically restart process if crashed (limit 3 times in 4 minutes)
respawn limit 3 240

# our process forks(clones) many times (maybe 10) so expect fork, or expect daemon are useless.
# we use the technique explained here http://stackoverflow.com/questions/12200217/can-upstart-expect-respawn-be-used-on-processes-that-fork-more-than-twice

# start the container in the pre-start script
pre-start script
    # wait (if necessary) for our docker context to be accessible
    while [ ! -f /projects/cups-server/docker-compose.yml ]
    do
      sleep 1
    done
    /usr/local/bin/docker-compose -f /projects/cups-server/docker-compose.yml up -d
end script

# run a process that stays up while our docker container is up. Upstart will track this PID
script 
    sleepWhileAppIsUp(){
        while docker ps | grep -q "$1"; do
            sleep 2
        done
    }

    sleepWhileAppIsUp "cupsserver_cups_"
end script

# stop docker container after the stop event has completed
post-stop script
    if docker ps | grep -q cupsserver_cups_;
    then
        /usr/local/bin/docker-compose -f /projects/cups-server/docker-compose.yml stop
    fi
end script

With this config, my services start automatically, and will respawn up to 3 times if they die. I can stop manually using sudo service cups-server stop and start again using sudo service cups-server start etc.

Make sure you start the docker container as a daemon (docker-compose ... up -d) or the pre-start script will not exit.