Since its launch bug reporting for packages sourced from the Ubuntu Cloud Archive for Ubuntu 12.04 LTS has been a little awkward and somewhat manual.
As of apport version 2.0.1-0ubuntu17.2, you can now:
ubuntu-bug <pkgname>
for packages from the Cloud Archive and bugs will get routed to the correct project in Launchpad with lots of extra bug data.
Thanks to those who spent time reporting bugs to-date – hopefully this will make you lives a little easier!
EOM
A number of people has asked about the availability of OpenStack 2012.2.1 in Ubuntu 12.10 and the Ubuntu Cloud Archive for Folsom; well its finally out!
Suffice to say it took longer that expected so we are making some improvements to the way we manage these micro-releases going forward which should streamline the process for 2012.2.3.
In order to help users and administrators of the Ubuntu Cloud Archive track which versions of what are where, the Ubuntu Server team are now publishing Cloud Archive reports for Folsom and Grizzly.
Grizzly g2 is currently working its way into the Ubuntu Cloud Archive (its already in Ubuntu Raring) and should finish landing into the updates pocket next week.
We now have Ceph fully integrated into the testing that we do around OpenStack; this picked up a regression in Nova and Cinder in the run up to 2012.2.1.
This highlights the value of the integration and system testing that we do in the Ubuntu OpenStack CI lab (see my previous post for details on the lab). Identifying regressions was high on the list of initial objectives we agreed for this function!
Focus at the moment is on enabling testing of Grizzly on Raring (its already up and running for Precise) and working on an approach to testing the OpenStack Charm HA work currently in-flight within the team. In full this will require upwards of 30 servers to test so we are working on a charm that deploys Ubuntu Juju and MAAS (Metal-as-a-Service) on a single, chunky server, allowing for physical-server-like testing of OpenStack in KVM. For some reason seeing 50 KVM instances running on a single server is somewhat satisfying!
This work will also be re-used for more regular, scheduled testing outside of the normal build-deploy-test pipeline for scale-out services such as Ceph and Swift – more to follow on this…
Ceilometer has also been added to the lab; at the moment we are build testing and publishing packages in the Grizzly Trunk PPA; Yolanda is working on a charm to deploy Ceilometer.
The next Ceph LTS release (Bobtail) is now available in Ubuntu Raring and the Ubuntu Cloud Archive for Grizzly.
One of the key highlights for this release is the support for Keystone authentication and authorization in the Ceph RADOS Gateway.
The Ceph RADOS Gateway provides multi-tenant, highly scalable object storage through Swift and S3 RESTful interfaces.
Integration of the Swift protocol with Keystone completes the complementing story that Ceph provides when used with OpenStack.
Ceph can fulfil ALL storage requirements in an OpenStack deployment; its integrated with Cinder and Nova for block storage, Glance for image storage and can now directly provide integrated, Swift compatible, multi-tenant object storage.
Juju charm updates to support Keystone integration with Ceph RADOS Gateway are in the Ceph charms in the charm store.
As Grizzly is about to release its first milestone, the Ubuntu Server Team thought it was a good opportunity to give an update on Ubuntu Server activities around Openstack.
Aside from a few SRU’s which are working through the system, the Folsom Cloud Archive for Ubuntu 12.04 is available and ready for use.
Please report any bugs that you find in packages from the Cloud Archive here.
In terms of communication around the Cloud Archive, general announcements about milestone and release availability will be made on ubuntu-cloud-anounce@lists.ubuntu.com.
We have also setup a new ML, cloud-archive-changes@lists.ubuntu.com, which will be higher volume, per upload notifications as new and updated packages land in the Cloud Archive:
For details on how to enable and use the Cloud Archive on Ubuntu 12.04 look here.
As we did for Folsom, a PPA is being maintained for Grizzly on Ubuntu 12.04 and Ubuntu Raring with the latest changes for each of the core Openstack components.
This PPA should also contain any required dependencies to support Grizzly on Ubuntu 12.04 – check out its Launchpad page for full details on use and current build status.
Please report bugs against Ubuntu for any issues that you find during development in packages from the Grizzly PPA.
Note that these packages are not supported in the same way as the Cloud Archive; so please don’t use them post release of Grizzly in your production deployments!
Packaging branches are maintained by the Openstack Ubuntu Testing team in the following branch URI format:
lp:~openstack-ubuntu-testing/<COMPONENT_NAME>/<UPSTREAM-RELEASE>
For example:
lp:~openstack-ubuntu-testing/nova/folsom lp:~openstack-ubuntu-testing/nova/grizzly lp:~openstack-ubuntu-testing/quantum/grizzly
If you have a packaging change that you would like to contribute, Launchpad merge proposals should be made against these branches.
Note that the grizzly branch for any given component feeds both the main development release of Ubuntu and the Cloud Archive for Ubuntu 12.04.
Please target ‘UNRELEASED’ in the changelog entry (rather than ‘raring’ or ‘quantal’ for example) unless you are the Ubuntu Server Dev responsible for the next upload to the Ubuntu or Cloud archive.
These branches feed the automated build and deployment testing of Openstack on Ubuntu; see Jenkins for results.
The Ubuntu Server team continues to focus on the quality of Openstack on Ubuntu; as part of this we undertake significant CI testing of Openstack on bare metal.
We are pleased to announce that the following components are currently being added to the Openstack CI reference architecture that we test in the lab:
We are also adding Ceph as an option, with support for Glance, Cinder and Nova.
As a result there may be periods of time when limited automated deployment and testing activity is undertaken whilst manual testing of these changes is underway in the lab.
Please refer to the Server Team Wiki for more details on branches, PPA’s and testing activities.
Ceph is a distributed storage and network file system designed to provide excellent performance, reliability, and scalability.
Sounds pretty cool right?
Ceph is forging the way in delivering petabyte/exabyte scale storage to thousands of clients using commodity hardware.
This post outlines some of the key activities that the Ubuntu Server Team have undertaken during the Ubuntu 12.10 development cycle to improve the Ceph experience on Ubuntu.
Ubuntu 12.10 features Ceph 0.48.2 ‘Argonaut’, the first release of Ceph with long-term support.
While development continues at a blistering pace and new releases will contain new features, the 0.48.x series will only receive critical bug-fixes and stability improvements.
This is a really important step for Ceph deployments; having a stable, supported release to baseline on is critical to the operation and stability of production environments.
For more information on the 0.48.x releases, see the release notes for Ceph.
For Ubuntu 12.04, Ceph was included in Ubuntu ‘main’ which means that it receives an increased level of focus from both the Ubuntu Server and Security teams (underwritten by Canonical) for the lifecycle of the Ubuntu release. However, to make this happen for the 12.04 release, some features of the packaging had to be disabled.
The good news is that those missing features have now been re-enabled in Ubuntu 12.10:
This re-aligns the Ubuntu packaging with the packages available directly from Ceph and in Debian.
Ceph can now be deployed effectively using Juju, the service orchestration tool for Ubuntu Server.
The Ceph charms for Juju build upon the automation work done by Tommi Virtanen from Inktank (who I think should win an award for his innovative use of Upstart for bootstrapping Ceph Object Storage Daemons).
The charms are still pending review for entry into the Juju Charm Store as the official charms but if you want to try them out:
cat > config.yaml << EOF ceph: fsid: ecbb8960-0e21-11e2-b495-83a88f44db01 monitor-secret: AQD1P2xQiKglDhAA4NGUF5j38Mhq56qwz+45wg== osd-devices: /dev/vdb /dev/vdc /dev/vdd /dev/vde ephemeral-unmount: /mnt EOF juju deploy -n 3 --config config.yaml --constraints="cpu=2" cs:~james-page/quantal/ceph
Some time later you should have a small three node Ceph cluster up and running. You can then expand it with further storage nodes:
cat >> config.yaml << EOF ceph-osd: osd-devices: /dev/vdb /dev/vdc /dev/vdd /dev/vde ephemeral-unmount: /mnt EOF juju deploy -n 3 --config config.yaml --constraints="cpu=2" cs:~james-page/quantal/ceph-osd juju add-relation ceph ceph-osd
And then add a RADOS Gateway for RESTful access:
juju deploy cs:~james-page/quantal/ceph-radosgw juju add-relation ceph ceph-radosgw juju expose ceph-radosgw
The ceph-radosgw charm can also be scaled-out and fronted with haproxy:
juju add-unit -n 2 ceph-radosgw juju deploy cs:precise/haproxy juju add-relation haproxy ceph-radosgw
You should now have a deployment that looks something like this (click to explode):
Note that the above examples assume that you have a Juju environment already configured and bootstrapped – if you have not read this.
The ceph and ceph-osd charms require additional block storage devices to work correctly so will not work with the Juju local provider; they have been tested in OpenStack, ec2 and MAAS environments and generally work OK (aside from one issue when ec2 instances get domU-XX hostnames).
All of the charms have README’s – take a look to find out more.
Credit to Paul Collins from the Canonical IS Projects team for initial work on the ceph charm.
OpenStack provides direct integration with Ceph in two ways:
Due to the scalable, resilient nature of Ceph, integration with OpenStack presents a compelling proposition.
Sebastien Han has already done a great job of explaining how to configure and use these features in OpenStack so I’m not going to go into the finer details here.
The OpenStack Juju charms for Ubuntu 12.10 will be updated to optionally use Ceph as a block and object storage back-end; here’s a preview:
juju add-relation glance ceph juju add-relation nova-volume ceph juju add-relation nova-compute ceph
Job done…
Ceph plans for the next Ubuntu release might include:
Follow the Ceph Blueprint and UDS-R session to see how this pans out.
During the Ubuntu precise development cycle the Canonical Platform Server Team have been working on automating testing of Openstack on Ubuntu.
The scope of this work was:
Openstack do a lot of pre-commit testing through the use of gerrit with Jenkins; we wanted to supplement this with Ubuntu focused testing to provide another dimension to the testing already completed upstream.
So grab a coffee and make yourself comfortable; this is not a short read….
Lab Setup
The Ubuntu Openstack QA lab consists of 12 servers; the primary server in the solution is an Ubuntu 11.10 install providing the following functions:
This server also acts at the gateway into and out of the Lab (it’s setup as a NAT router).
The other 11 servers are registered in Cobbler; All servers are connected to a Sentry CDU (Cabinet Distribution Unit) which allows full power control from Cobbler – thanks goes to Andres Rodriguez for developing the required fence component for Cobbler to support this type of CDU.
Preseeded LVM Snapshot Installs
To initiate a new integration test run requires all machines to be powered down and re-provisioned from scratch. It is essential that our deployment and test runs can cope the frequency of upstream commits, particularly as the frequency increases as Openstack approaches milestones and releases. After getting the initial lab setup in place, we were able to tear down all machines, re-provision and deploy Openstack in ~30mins.
It was important that we are able to minimize the time taken to complete the testing cycle. To do so, we’ve employed the use of LVM snapshotting and restoration of the root partition during the the netboot installation. The process is as follows:
This process takes place on all nodes in parallel. With it in place, we were able to cut down the time it took to tear-down and re-provision a node from ~30 minutes to 10 to 15 minutes depending on the service being deployed.
By taking this approach we are also minimize the chance of any nodes hitting an archive inconsistency during installation. This is a known issue when deploying the development release and halts installation on any node that hits it, failing the entire deployment.
All of this is embedded in debian-installer preseeds via Cobbler snippets. The snippets and kick starts are available at lp:~openstack-ubuntu-testing/+junk/cobbler-lvm-snapshot.
In the future, we’ll be investigating the use of kexec as an alternative to reboot after snapshot restoration to reduce the time spent waiting on servers to boot. This should minimize the test cycle even more. Credit to James Blair for the idea (see http://amo-probos.org/post/11).
Management of Jenkins
All of the projects in Jenkins are managed using Jinja2 XML templates in-conjunction with python-jenkins (python-jenkins); this makes it really easy to setup new jobs in the lab and reconfigure existing ones as required (as well as providing great backup!).
Templates and management scripts can be found in lp:~openstack-ubuntu-testing/+junk/jenkins-qa-lab
Testing Openstack Essex on Ubuntu Precise
This testing was the first to be setup in the lab. Jenkins (using the git plugin) monitors the upstream github.com repositories for commits on the master branch. When a change is detected the following process is triggered:
Build
Objective: Validate that upstream trunk still builds OK with current packaging for Ubuntu.
On the assumption that the package built OK locally:
This process is managed by a single script (tarball.sh); Credit to Chuck Short for pulling together this part of the process based on work from Openstack upstream.
For changes to the nova project the deploy phase is then executed.
Deploy
Objective: Validate that packages install, can be configured and reach a know good state prior to execution of testing.
This phase of testing uses Juju with Cobbler to deploy Openstack into the QA lab infrastructure; It utilizes branches of the Openstack charms to support use of a local archive along with a deployer wrapper around Juju written by Adam Gandelman which executes the actual deployment using Juju and monitors for errors.
The deployer is configured to know where to get the right codebase for the Openstack charms, which services to deploy and which relations to setup between services. As you can see from the above diagram this is non-trivial but the charms and Juju do most of the hard work.
Once Openstack is deployed successfully the test phase is then executed.
Test
Objective: Validate that the Openstack deployment in the lab actually works!
At this point, we can run any integration tests we wish against the newly deployed cloud. This testing is able to help us achieve multiple goals:
At the moment this phase looks like this:
Note: These are the same sets of tests that are currently run against proposed commits to gerrit upstream.
Longer term we aim to use the Openstack Tempest test suite in the lab; Adam is currently working on getting this up and running.
Reporting
The Jenkins instance in the QA lab is not publicly accessible; however all jobs run in the lab are published out (using the Jenkins build-publisher plugin) to http://jenkins.qa.ubuntu.com so that people can see the current state of the testing packaging in Ubuntu precise.
We are also working on setting up email notifications.
Success so far
Juju charms deploy Openstack components in a configuration that is compatible with upstream trunk prior to updates to packaging in Ubuntu. Previously packages were updated in the archive first while Juju charm updates lagged behind as incompatibilities were uncovered after the fact.
We enabled automated testing 2 days prior to the 3rd Essex milestone release. We were able to uncover and help fix a handful of bugs upstream before the release, including critical bugs like 921784. In the past, these bugs were typical uncovered after the release (both upstream and in Ubuntu).
Since E3, there have been even more critical bugs uncovered by this testing and fixed upstream, some of which are only applicable to Ubuntu-specific configurations (not tested upstream) and would have been uncovered by users after code hit the Ubuntu archive (See 922232).
Further Plans for the Lab
Pre-commit testing of changes to stable branches; The Ubuntu Server team are working upstream on maintaining the stable branches of released versions of OpenStack – this work will validate patches proposed to stable branches in review.openstack.org against the current version of the packaging in released versions of Ubuntu. Initially this will target Diablo on Ubuntu 11.10 but will also support Essex on Ubuntu 12.04 once released. Ideally the testing process will provide feedback on review.openstack.org to help the stable release team review proposed patches.
References
Jenkins job configurations: lp:~openstack-ubuntu-testing/+junk/jenkins-qa-lab
Scripts supporting the lab: lp:~openstack-ubuntu-testing/+junk/jenkins-scripts
LVM snapshot preseeds and Cobbler snippets: lp:~openstack-ubuntu-testing/+junk/cobbler-lvm-snapshot
All other relevant scripts, charm branches, etc: https://code.launchpad.net/~openstack-ubuntu-testing/
Credits
Overall management of delivery and general whip cracking: Dave Walker
Lab installation and base configuration: Pete Graner, Tim Gardner, Brad Figg, James Page
Fence agent for network power control of servers: Andres Rodriguez
Source package creation and build process: Chuck Short and James Page
Deployment testing using Juju: Adam Gandelman
Testing of Openstack: Adam Gandelman
Jenkins packaging, configuration and management: James Page
Gerrit Plugin for pre-commit testing and generally great ideas: Monty Taylor and James Blair
Writing and reviewing this post: Adam Gandelman, Chuck Short and Dave Walker.
