The scope of this work was:

1. Per-commit testing of Openstack trunk to evaluate the current state of the upstream codebase in-conjunction with the current packaging in Ubuntu precise and the current Juju charms to deploy Openstack.
2. SRU testing for Openstack Diablo on Ubuntu 11.10.

Openstack do a lot of pre-commit testing through the use of gerrit with Jenkins; we wanted to supplement this with Ubuntu focused testing to provide another dimension to the testing already completed upstream.

So grab a coffee and make yourself comfortable; this is not a short read….

Lab Setup

The Ubuntu Openstack QA lab consists of 12 servers; the primary server in the solution is an Ubuntu 11.10 install providing the following functions:

1. Juju – used to deploy Openstack charms in the Lab
2. Cobbler to support server provisioning (using the Ubuntu Orchestra packages in Oneiric)
3. Jenkins CI – provides triggering based on upstream commits to github repositories and general job control and reporting.
4. Schroots for Oneiric and Precise for building packages locally
5. A reprepro managed local archive for Oneiric and Precise
6. Squid based archive caching to reduce installation times in the lab

This server also acts at the gateway into and out of the Lab (it’s setup as a NAT router).

The other 11 servers are registered in Cobbler; All servers are connected to a Sentry CDU (Cabinet Distribution Unit) which allows full power control from Cobbler – thanks goes to Andres Rodriguez for developing the required fence component for Cobbler to support this type of CDU.

Preseeded LVM Snapshot Installs

To initiate a new integration test run requires all machines to be powered down and re-provisioned from scratch.  It is essential that our deployment and test runs can cope the frequency of upstream commits, particularly as the frequency increases as Openstack approaches milestones and releases.   After getting the initial lab setup in place, we were able to tear down all machines, re-provision and deploy Openstack in ~30mins.

It was important that we are able to minimize the time taken to complete the testing cycle.   To do so, we’ve employed the use of LVM snapshotting and restoration of the root partition during the the netboot installation.   The process is as follows:

1. Test run begins
2. Juju deploys a service (i.e. nova-compute)
3. A machine is netbooted and a preseeded LVM-based Ubuntu installation takes place onto /dev/qalab/root
4. At the end of the installation, the root filesystem is moved to /dev/qalab/pristine-[release]-root and a snapshot created at /dev/qalab/root
5. The machine reboots, runs Juju and deploys nova-compute as pat of the rest of the Openstack deployment. This deployment is smoke tested.
6. The next test run begins.  All machines are terminated. Juju redeploys nova-compute, a machine is netbooted and Ubuntu installation kicks off.
7. The installation checks for the existence of a logical volume at /dev/qalab/pristine-[release]-root.  If it exists, it creates a new snapshot at /dev/qalab/root and reboots. If it does not, continues with installation and goto step 4.
8. System reboots, Juju installs and redeploys nova-compute to a fresh Ubuntu installation.

This process takes place on all nodes in parallel.  With it in place, we were able to cut down the time it took to tear-down and re-provision a node from ~30 minutes to 10 to 15 minutes depending on the service being deployed.

By taking this approach we are also minimize the chance of any nodes hitting an archive inconsistency during installation. This is a known issue when deploying the development release and halts installation on any node that hits it, failing the entire deployment.

All of this is embedded in debian-installer preseeds via Cobbler snippets.  The snippets and kick starts are available at lp:~openstack-ubuntu-testing/+junk/cobbler-lvm-snapshot.

In the future, we’ll be investigating the use of kexec as an alternative to reboot after snapshot restoration to reduce the time spent waiting on servers to boot.  This should minimize the test cycle even more. Credit to James Blair for the idea (see http://amo-probos.org/post/11).

Management of Jenkins

All of the projects in Jenkins are managed using Jinja2 XML templates in-conjunction with python-jenkins (python-jenkins); this makes it really easy to setup new jobs in the lab and reconfigure existing ones as required (as well as providing great backup!).

Templates and management scripts can be found in lp:~openstack-ubuntu-testing/+junk/jenkins-qa-lab

Testing Openstack Essex on Ubuntu Precise

This testing was the first to be setup in the lab.  Jenkins (using the git plugin) monitors the upstream github.com repositories for commits on the master branch.  When a change is detected the following process is triggered:

Build

Objective: Validate that upstream trunk still builds OK with current packaging for Ubuntu.

1. A new snapshot upstream tarball is generated based on the latests commit to the upstream component.
2. The latest archive packaging for the component is pulled in from lp:~ubuntu-server-dev/<COMPONENT>/essex
3. Any changes in the testing packaging for the component are merged from lp:~openstack-ubuntu-testing/<COMPONENT>/essex
4. New changelog entries are automatically created for the new upstream commits.
5. The source package is generated and built in a clean schroot using sbuild locally.

On the assumption that the package built OK locally:

1. The source package is uploaded to the Testing PPA (ppa:openstack-ubuntu-testing/testing)
2. The testing packaging branch is push back to lp:~openstack-ubuntu-testing/<COMPONENT>/essex.
3. The binary packages from the sbuild are installed into the local reprepro managed archive.

This process is managed by a single script (tarball.sh); Credit to Chuck Short for pulling together this part of the process based on work from Openstack upstream.

For changes to the nova project the deploy phase is then executed.

Deploy

Objective: Validate that packages install, can be configured and reach a know good state prior to execution of testing.

This phase of testing uses Juju with Cobbler to deploy Openstack into the QA lab infrastructure; It utilizes branches of the Openstack charms to support use of a local archive along with a deployer wrapper around Juju written by Adam Gandelman which executes the actual deployment using Juju and monitors for errors.

The deployer is configured to know where to get the right codebase for the Openstack charms, which services to deploy and which relations to setup between services. As you can see from the above diagram this is non-trivial but the charms and Juju do most of the hard work.

Once Openstack is deployed successfully the test phase is then executed.

Test

Objective: Validate that the Openstack deployment in the lab actually works!

At this point, we can run any integration tests we wish against the newly deployed cloud.  This testing is able to help us achieve multiple goals:

• Early detection of upstream bugs that break Openstack functionality on Ubuntu
• Verification that packaging branches in the development version of Ubuntu are compatible with upstream trunk.
• Using these packages, verification that our Juju charms are deploying a functional Openstack cloud and are up-to-date with any deployment-related configuration changes upstream.

At the moment this phase looks like this:

1. Configure the Openstack deployment (Adams deployer script provides some utility functions for locating specific services in the environment)
   • Creates network configuration in Nova for the private instance network as well as a pool of public floating IPs.
   • Upload an image into the Glance server for use during testing
   • Creates EC2 credentials in the Keystone server for use during testing.
2. Run the devstack exercise test scripts which ensure basic functionality of the deployment. Currently, this includes:
   • Basic euca-tools EC2 API for starting and stopping instances
   • EC2 AMI bundle uploads
   • Floating IP allocation, association and connectivity to instance
   • Volume creation and attachment to instance

Note: These are the same sets of tests that are currently run against proposed commits to gerrit upstream.

Longer term we aim to use the Openstack Tempest test suite in the lab; Adam is currently working on getting this up and running.

Reporting

The Jenkins instance in the QA lab is not publicly accessible; however all jobs run in the lab are published out (using the Jenkins build-publisher plugin) to http://jenkins.qa.ubuntu.com so that people can see the current state of the testing packaging in Ubuntu precise.

We are also working on setting up email notifications.

Success so far

Juju charms deploy Openstack components in a configuration that is compatible with upstream trunk prior to updates to packaging in Ubuntu.  Previously packages were updated in the archive first while Juju charm updates lagged behind as incompatibilities were uncovered after the fact.

We enabled automated testing 2 days prior to the 3rd Essex milestone release.  We were able to uncover and help fix a handful of bugs upstream before the release, including critical bugs like 921784.  In the past, these bugs were typical uncovered after the release (both upstream and in Ubuntu).

Since E3, there have been even more critical bugs uncovered by this testing and fixed upstream, some of which are only applicable to Ubuntu-specific configurations (not tested upstream) and would have been uncovered by users after code hit the Ubuntu archive (See 922232).

Further Plans for the Lab

Pre-commit  testing of changes to stable branches;  The Ubuntu Server team are  working upstream on maintaining the stable branches of released versions  of OpenStack – this work will validate patches proposed to stable  branches in review.openstack.org against the current version of the  packaging in released versions of Ubuntu.  Initially this will target  Diablo on Ubuntu 11.10 but will also support Essex on Ubuntu 12.04 once  released.  Ideally the testing process will provide feedback on  review.openstack.org to help the stable release team review proposed  patches.

References

Jenkins job configurations: lp:~openstack-ubuntu-testing/+junk/jenkins-qa-lab

Scripts supporting the lab: lp:~openstack-ubuntu-testing/+junk/jenkins-scripts

LVM snapshot preseeds and Cobbler snippets: lp:~openstack-ubuntu-testing/+junk/cobbler-lvm-snapshot

All other relevant scripts, charm branches, etc: https://code.launchpad.net/~openstack-ubuntu-testing/

Credits

Overall management of delivery and general whip cracking: Dave Walker

Lab installation and base configuration: Pete Graner, Tim Gardner, Brad Figg, James Page

Fence agent for network power control of servers: Andres Rodriguez

Source package creation and build process: Chuck Short and James Page

Deployment testing using Juju: Adam Gandelman

Testing of Openstack: Adam Gandelman

Jenkins packaging, configuration and management: James Page

Gerrit Plugin for pre-commit testing and generally great ideas: Monty Taylor and James Blair

Writing and reviewing this post: Adam Gandelman, Chuck Short and Dave Walker.

I expect to upload another version in the next few weeks to enable the SSH command line interface to Jenkins (requires some more Jenkins tool-chain work to get everything building from source).

Huge thanks go to Tony Mancill and Damien Raude-Morvan who have sponsored my Jenkins packaging work in Debian!

I’ll sync this new version over to Ubuntu precise (and push into the new Jenkins Backports PPA for Ubuntu 11.10) once alpha 2 is out of the door…

Python Jenkins provides Python bindings to the Jenkins Remote API.

I’m pleased to announce that Python Jenkins 0.2 is available in Ubuntu Oneiric Ocelot and the project has now migrated to Launchpad for bug tracking, version control and release management.

The 0.2 release includes a number of bug fixes and new methods for managing Jenkins slave node configuration remotely – this is already being used in the juju charm for Jenkins to automatically create new slave node configuration when slave node units are added to a deployed juju environment (blog posting to follow….)

A quick overview…

It’s pretty easy to get started with python-jenkins on the latest Ubuntu development release – if you are running an earlier release of Ubuntu then you can use the PPA (Natty only ATM but more to follow):

sudo apt-get install python-jenkins

The library is also published to PyPI so you can use pip if you are not running Ubuntu.

Here’s a quick example script:

#!/usr/bin/python
import jenkins
# Connect to instance - username and password are optional
j = jenkins.Jenkins('http://hostname:8080', 'username', 'password')

# Create a new job
if not j.job_exists('empty'):
    j.create_job('empty', jenkins.EMPTY_CONFIG_XML)
j.disable_job('empty')

# Copy a job
if j.job_exists('empty_copy'):
    j.delete_job('empty_copy')
j.copy_job('empty', 'empty_copy')
j.enable_job('empty_copy')

# Reconfigure an existing job and build it
j.reconfig_job('empty_copy', jenkins.RECONFIG_XML)
j.build_job('empty_copy')

# Create a slave node
if j.node_exists('test-node'):
    j.delete_node('test-node')
j.create_node('test-node')

You can find the current documentation here.

Happy automating!

You can see the results here (limited time offer only); and here are the details of how I setup this rebuild.

Setup your rebuild server

So first things first; the current development release of Ubuntu (Oneiric Ocelot) includes the most recent LTS release of Jenkins so installing its pretty easy as long as you are using this release:

sudo apt-get install jenkins


You probably want to configure Jenkins with an administrator email address and point it at a mail relay so that Jenkins can tell you about failures.

Next you need to install a few development tools to make it all hang together nicely:

sudo apt-get install ubuntu-dev-tools sbuild apt-cacher-ng

This should get you to the point where you can setup the jenkins account to use sbuild:

sudo usermod -G sbuild jenkins
sudo stop jenkins
sudo start jenkins

Create sbuild environments

I decided that I wanted to compare current build results with openjdk-6 against the rebuild with openjdk-7 so I created two sbuild environments on my server – one for openjdk-6 and one for openjdk-7:

mk-sbuild --name=oneiric-java-6 oneiric
mk-sbuild --name=oneiric-java-7 oneiric


You might have noticed that I also installed apt-cacher-ng – as alot of the dependencies that get downloaded are the same having a local apt cache makes alot of sense to speed things up; its pretty easy to point the schroots we just created at the local instance of apt-cacher-ng:

sudo su -c "echo 'Acquire::http { Proxy \"http://localhost:3142\"; };' > /var/lib/schroot/chroots/oneiric-java-6-amd64/etc/apt/apt.conf.d/02proxy"
sudo su -c "echo 'Acquire::http { Proxy \"http://localhost:3142\"; };' > /var/lib/schroot/chroots/oneiric-java-7-amd64/etc/apt/apt.conf.d/02proxy"

You will also need to setup some self signed keys to make sbuild work:

sudo sbuild-update -k

So we now have two sbuild environments setup – but we need one to point at openjdk-7 instead of openjdk-6 – I stuck a modified copy of java-common (which determines the default-jdk) in a PPA:

sudo schroot -c oneiric-java-7-amd64-source
apt-get install python-software-properties
add-apt-repository ppa:james-page/default-jdk-7

Obviously this could just as easily be an upgrade to mysql, postgresql or any other package of your choosing.

Setting up Jenkins

I split this rebuild into ‘main’ and ‘universe’ components and used a mutli-configuration project with axis for ‘package’ (containing the list of packages I wanted to rebuild) and ‘version’ (java-6, java-7); I’ve included the config.xml for one of the projects here.  You can also see the configuration here.

The build is a very simple script:

#!/bin/bash
export HOME=/var/lib/jenkins
rm -Rf *
pull-lp-source ${package}
sbuild -d oneiric-${version} -A -n ${package}*.dsc

That should do the trick; you can then submit the project for build and it will gradually churn through the packages emailing you with any failures.

Getting Started

If you are not using the development release of Ubuntu, Oneiric Ocelot, grab a copy of the latest alpha-3 milestone pre-release:

• ISO Images: http://cdimages.ubuntu.com/releases/oneiric/alpha-3/
• Cloud Images: http://cloud-images.ubuntu.com/releases/oneiric/alpha-3/

Note that alpha releases are NOT always stable – be careful!  Jenkins is quite happy running on either a Desktop or a Server so take your pick.

Once you have an install of Oneiric Ocelot…

apt-get install jenkins


…should get you going.  Jenkins should startup and be visible on http://localhost:8080 (or the name of your server if you are installing it elsewhere).

A few things to note…

The package for Ubuntu differs in a few ways from the upstream Jenkins .deb packages you might have been using to-date:

• Plugins are not bundled by default – normally Jenkins ships with the following plugins:
  • SSH Slaves Plugin
  • Maven Integration Plugin
  • Subversion Plugin

  Plugins should be compatible with this distribution packaged version of Jenkins – these can be installed through the ‘Manage Plugins’ option within Jenkins as normal.  I would recommend installing the Maven Integration Plugin at a minimum.

• Native OS integration features that are disabled:
  • Solaris: support for libzfs and libembeddedsu4j.
  • Windows: native control of remote windows slaves, native integration with Windows process management, windows service deployment of Jenkins

  It is still possible to use Windows slaves; however they will need to be launched using JNLP from the slave itself.

• Branding: if available on the client, the Web UI will use the Ubuntu font.
• Packages + wrapper scripts are provided for:
  • Monitoring of arbitrary jobs: jenkins-external-job-monitor + see man jenkins-monitor-job for more details.
  • Jenkins CLI: jenkins-cli + see man jenkins-cli for more details.

Using Jenkins

The Jenkins website has a great getting started guide – see https://wiki.jenkins-ci.org/display/JENKINS/Use+Jenkins.

Obviously you can skip the ‘Installing Jenkins’ section.

Reporting Bugs

If you find any problems please report bugs here https://bugs.launchpad.net/ubuntu/+source/jenkins

You can also find me (jamespage) in #ubuntu-server on Freenode IRC most weekdays.

Enjoy…

libakuma-java: missing-classpath libjna-java


This is a new check that validates that the jar files associated with a debian -java package have a Class-Path entry set in the META-INF/MANIFEST.mf file.

This is used by wrapper scripts to dynamically set the classpath for Java applications at runtime based on the libraries that they use.

However, if you are packaging Maven based projects you don’t get this set for free (although that would be a great feature for maven-debian-helper).

Luckily the javahelper package has a CDBS class that can help out; simply add javahelper to the Build-Depends for your package then follow these steps:

1) debian/rules

Add the javahelper.mk class into the make file:

#!/usr/bin/make -f

...
2) Add a .classpath file for the binary package

In my case this gets dropped into debian/libakuma-java.classpath

usr/share/java/akuma.jar /usr/share/java/jna.jar

Rebuild your package; the jar files should now have Class-Path entries and you should be lintian clean.

Enjoy.

I’ve not attended a Puppet Camp before; I went with high expectations based on feedback from friends and colleagues and I was not disappointed. The event is a great mix of presentations from the Puppet Labs team, Puppet users and community members with break-out ‘Unconference’ sessions in the afternoon to focus on specific areas of interest that we all voted on after lunch.

Keynote

The conference started off with the keynote delivered by Luke Kanies, the founder of both Puppet and Puppet Labs. Puppet 2.7 is about to hit RC2 and Luke presented a great overview of what we can expect to see in this release. The focus of this release has been in the following areas:

• Static Compiler Plugin; compiles file resources into the catalogue which will reduce the amount of calls between Puppet agents and the puppet master improving overall efficiency.
• Certificate API: Puppet CA gets a new RESTful remote API for management of certificates.  This will be really important for integration into provisioning systems both in the data centre and in the cloud.
• Puppet Faces; The puppet CLI will provide a new set of ‘Faces’ commands which allow more direct interface to Puppet internals.  This will allow uses to build custom agent behaviour for bespoke solutions much more easily.
• Minimising Puppet Core: removal of excess dependencies (such as nagios plugins) to make the core of Puppet as simple and as efficient and possible while still allowing easy extension as required.

The other key change in 2.7 is a shift from GPL to Apache based licensing for the product. This will appease enterprises who don’t like/won’t use GPL licensed software and will allow Puppet Labs to side-step around some of the unproven concepts of whether an extension developed in Ruby is considered a change to core product under GPL – no one really seems to know the implications of this uncertainty.

Other stuff that Puppet Labs are thinking about for the future include:

• Dealing with localised change in the Puppet agent to make it easier to feed back into central manifests and templates.
• Cross-node application configuration management; Puppet is great at managing individual nodes but not so great at orchestrating change in a synchronised way across multiple nodes.  I suspect mCollective will be part of this story…
• Database support
• Change lifecycle management; mcollective seems to be key in this strategy but its not exactly clear how yet.
• Further language changes to separate code from data

All good stuff IMHO.

Extending Puppet

Richard Crowley presented briefly about DevStructure’s current project, Blueprint, a tool to reverse engineer server configuration into Git.

He also presented on a number of ways in  which Puppet can be extended to-do interesting and crazy stuff – it was a good overview into the internal workings of Puppet (even if some of the examples came with warnings).  One of the neatest concepts was storing additional server configuration meta-data in DNS TXT records which puppet could then retrieve and use when compiling catalogues – kind of like a DNS based external node classifier.

Working with Puppet Modules

Henrik Lindberg from CloudSmith presented on some best practices for developing Puppet Modules and some of the work that is currently ongoing around the Puppet Forge, a central repository for community developed Puppet Modules.

He demonstrated Geppetto, an Eclipse based IDE for developing Puppet Modules; I think that developments such as this are key in the whole DevOps picture as it gives Developers a toolset that they should find familiar which allows then to develop functionality that can then be used for system configuration in Puppet – after all Puppet is really about coding not writing configuration files…

Geppetto could be a good target for inclusion into Debian and Ubuntu in a similar style to PyDev (see eclipse-pydev) and would make a nice, pre-integrated package for Ubuntu Desktops.

Puppet Continuous Integration

Nokia have taken with whole ‘Puppet is Code’ development principle one step further by applying Continuous Integration techniques to the modules that they develop to manage the infrastructure that supports their location based services.

Oliver Hookins from Nokia presented on how they have used Jenkins and cucumber-puppet to automate behavioural driven testing of Puppet catalogue definitions prior to automated deployment into their production environment.  I really like this concept as it injects a great software development discipline into configuration management development.

Puppi

Alessandro Franceschi from Lab42 gave a couple of great demonstrations (complete with cheezy 80′s style synth music) of puppi, a Puppet module and CLI toolset to support the deployment of full applications and batch operations using Puppet manifests. Although this was really interesting in its own right, he also demo’ed mc-puppi – puppi integrated with mcollective – deploying applications across a large number of systems using mcollective. Its this type of integration that really demonstrates the power of broadcast based systems management through mcollective

Puppet DSL

Randall Hansen has been employed by Puppet Labs specifically to address the usability of puppet across all of its interfaces; CLI, DSL and Dashboard. Current focus is on making the CLI consistent and usable with actionable error messaging being a key focus. The Puppet CLI has moved towards a “puppet command subcommand” structure over the last couple of releases and it looks like this strategy will continue to be developed.

This session generated quite alot of questions including ‘when will puppet get its own shell?’ (on the roadmap but not being actively developed) and ‘plugin support for dashboard?’ (that sounds like a really cool idea).

Unconference

I attended various Unconference sessions over the two days of Puppet Camp – here are my key takeouts:

• Provisioning/Bootstrapping: an interesting session with respect to Ubuntu Orchestra as alot of the topics discussed where relevant to this project. Initial discussion focussed around which tools people where using for provisioning – Cobbler is in use (but seen as to specific to RHEL) but Foreman had alot more focus and interest. This is probably due to its tighter integration with Puppet; it can act as an external node classifier and can manage certificates during deployment, and its smart proxy architecture which looks really interesting (it can also proxy Puppet CA calls). The option to have a central management ‘foreman’ with deployment proxies in locations across the data centre(s) will fit well with what Enterprises want in terms of security so I can see its appeal.
• mCollective: demo and questions with R.I. Pienaar (project founder). I really like the re-use of MoM concepts in System Adminstration; it scales so much better than traditional approaches and makes administering a large number of servers a piece of cake. I think it will be really interesting to see how Puppet Labs position this project alongside Puppet and what the blueprint deployment will look like in 6-12 months time. Again this is a really important development to consider in the context of Ubuntu Orchestra.
• Network Device management; Puppet 2.7 has a feature which allows Cisco switches to be managed as nodes – this is the first step in this direction for Puppet. This is seen as a key development by the user base as it will support more integrated data centre configuration management across disciplines.

Summary

Puppet continues to develop strongly; the product and the company behind it are really maturing. I think there are a few key changes to integrate in the next few releases; specifically separation of data and code in the Puppet DSL and integration of mCollective into the Puppet Labs vision.

The community around Puppet continues to be very active; CloudSmith are really leading the charge with community developed modules and the user base seems to be very highly engaged with the product.

After just over 3 months I’ve managed to get a 99% built from source Jenkins into the Hudson Ubuntu Testing PPA; this is core Jenkins only at this point in time; no plugins come bundled (although you can still install them using the Plugin Manager).

Note that the PPA only provides packages for the Natty Narwhal development release.

If you want to help out with testing, read on….

Getting started

So first things first; lets add the Jenkins testing PPA and install the jenkins package:

sudo add-apt-repository ppa:hudson-ubuntu/testing
sudo apt-get update
sudo apt-get install jenkins

Wait some time and Jenkins should be up and running.

Accessing Jenkins

Once the package has installed you should be able to access Jenkins on http://localhost:8080 (or whatever server you installed it on).

Jenkins comes completely empty; no jobs, minimal configuration, no security.  Follow the normal Jenkins ‘Use Jenkins‘ guide to get started.

Reporting problems

This is a testing package so it ain’t going to be perfect. Please join the hudson-ubuntu-users team and subscribe to the mailing list on launchpad to discuss general issues and report problems.

You can also find me (jamespage) in #ubuntu-server on Freenode if you like to use IRC.

Whats still todo?

Well as I said this is 99% built from source; a few packages are either binary packages (built from jars) or packages that bundle their dependencies in the upstream source bundle. I’m working with upstream(s) to get these issues sorted.

That said, everything that is installed on your server or desktop in the jenkins package IS built from source. Offending packages are either used for testing or generating documentation during library builds.

I’m also going to start working on packaging a few core plugins from source (see the associated blueprint for details); for the time being you can download and install from within Jenkins. I will also create a patch so that this feature can be disable if required.  Please feel free to apply to join the team if you want to package a plugin.

The Future

Well ultimately probably the main Ubuntu archive and Debian; but lets leave that until next release….

And if you were wondering, Hudson renamed itself to Jenkins in late January (hence the Hudson name keeps creeping into this post).

Scott Moser has done some great work on automating this previously using a small framework of scripts. We discussed refactoring to use Python and boto to make the testing a bit easier to maintain and extend going forwards. This also allows re-use the existing Python unittest + subunit approach from the ISO testing project for testing each ec2 instance as it was started.

An initial test release of the code is available for use; you are going to need an AWS account to use this codebase and be aware that running large tests does cost quite alot of $$/££ so be careful!

Getting started

So the first thing you will need todo is install the ubuntu-server-ec2-testing package; this is available in a PPA and is currently only available for the Natty release:

sudo add-apt-repository ppa:ubuntu-server-ec2-testing-dev/testing
sudo apt-get update
sudo apt-get install ubuntu-server-ec2-testing


This installs the base ec2 testing package onto your machine.

Grab some test cases

Test case definitions and configurations are stored in a bzr branch on Launchpad; this allows configuration and the test itself to be managed outside of the packaged code.

bzr branch lp:~ubuntu-server-ec2-testing-dev/+junk/ec2-automated-tests tests

Take a look in the tests directory this creates:

• the default folder contains the test configuration for the ‘default’ test. This includes the cloud-init configuration that will be injected into each ec2 instance and the Python unittest file which tests the instance.
• *.yaml – these are example test suite configuration files; in this example we will use the test-micro.yaml example file

Setup your ec2 credentials

To access ec2 a key and secret is required; this is stored in another yaml file (~/.ec2.yaml):

AWS_ACCESS_KEY_ID: your access key id
AWS_SECRET_ACCESS_KEY: secret associated with key

Credentials can be setup from your account screens in Amazon Web Services.

Run a test suite

The ubuntu-server-ec2-testing package provides two scripts to execute tests:

• run-ubuntu-ec2-test: runs a single test case; run without any parameters for help.
• run-ubuntu-ec2-multi-test: runs a suite of test cases for multiple sizes, root storage options, regions and availability zones.

So in this example we are going to run the test-micro.yaml test suite.  The format should be fairly intuitive:

This suite is a) for maverick b) for the server (desktops later!) c) will run 1 instance of each TestCase d) will run the i386 image on a t1.micro e) in eu-west-1a and eu-west-1b and f) using ebs root filesystem storage. In total this will cost 2 x t1.micro hours (one for each instance).

To execute this test (against the latest maverick published images using the default test):

run-ubuntu-ec2-multi-test -t tests -f tests/test-micro.yaml default

After a short period of time (about 5 minutes) the test suite should have completed. In the current working directory you will find test results (in JUnit XML format) and console output under the eu-west-1a and eu-west-1b folders.

Part 2: integration with Jenkins to follow….

I had been getting by with a old 100MBps switch connected to my iMac which acted as a gateway onto the main wireless network in the house for the infrastructure that I use for testing Ubuntu Server; however the iMac gateway kept disabling itself and the performance was not great.

So last weekend I bit the bullet and ordered myself a pair of ZyXEL powerline adapters and a Netgear Gigabit network switch from e-buyer.

By 1245 on Monday I had a new home network – the setup was so easy it was ridiculous;  the powerline adapters automatically connected to each other as soon as I plugged them in and the Netgear switch was more than happy to chat to my BT HomeHub 2 through them with zero config.

Now that my office is completely hard-wired to my Internet connection latency seems to be much better (think that my BT HomeHub was a little overloaded) and the gigabit connectivity between devices is certainly helping with performance (I run a local Ubuntu archive cache using apt-cacher-ng to avoid using my Internet connection as much as possible).

Shame I only have a 5.5MBps Internet connection; and living in rural Norfolk I can’t see BT Infinity reaching me anytime soon…