Jenkins 1.424.2 in Debian unstable!

Just a short post to say that Jenkins 1.424.2 has landed in Debian; 1.409.3 was accepted a few weeks ago and I just uploaded 1.424.2 to unstable.

I expect to upload another version in the next few weeks to enable the SSH command line interface to Jenkins (requires some more Jenkins tool-chain work to get everything building from source).

Huge thanks go to Tony Mancill and Damien Raude-Morvan who have sponsored my Jenkins packaging work in Debian!

I’ll sync this new version over to Ubuntu precise (and push into the new Jenkins Backports PPA for Ubuntu 11.10) once alpha 2 is out of the door…

Package rebuild testing using Jenkins and sbuild

I’ve been working on rebuilding the Java packages in the Ubuntu archive that depend on ‘default-jdk’ against OpenJDK 7 to see where we are likely to have issues if and when we decide to upgrade the default version of Java in Ubuntu to OpenJDK 7

You can see the results here (limited time offer only); and here are the details of how I setup this rebuild.

Setup your rebuild server

So first things first; the current development release of Ubuntu (Oneiric Ocelot) includes the most recent LTS release of Jenkins so installing its pretty easy as long as you are using this release:

sudo apt-get install jenkins


You probably want to configure Jenkins with an administrator email address and point it at a mail relay so that Jenkins can tell you about failures.

Next you need to install a few development tools to make it all hang together nicely:

sudo apt-get install ubuntu-dev-tools sbuild apt-cacher-ng

This should get you to the point where you can setup the jenkins account to use sbuild:

sudo usermod -G sbuild jenkins
sudo stop jenkins
sudo start jenkins

Create sbuild environments

I decided that I wanted to compare current build results with openjdk-6 against the rebuild with openjdk-7 so I created two sbuild environments on my server – one for openjdk-6 and one for openjdk-7:

mk-sbuild --name=oneiric-java-6 oneiric
mk-sbuild --name=oneiric-java-7 oneiric


You might have noticed that I also installed apt-cacher-ng – as alot of the dependencies that get downloaded are the same having a local apt cache makes alot of sense to speed things up; its pretty easy to point the schroots we just created at the local instance of apt-cacher-ng:

sudo su -c "echo 'Acquire::http { Proxy \"http://localhost:3142\"; };' > /var/lib/schroot/chroots/oneiric-java-6-amd64/etc/apt/apt.conf.d/02proxy"
sudo su -c "echo 'Acquire::http { Proxy \"http://localhost:3142\"; };' > /var/lib/schroot/chroots/oneiric-java-7-amd64/etc/apt/apt.conf.d/02proxy"

You will also need to setup some self signed keys to make sbuild work:

sudo sbuild-update -k

So we now have two sbuild environments setup – but we need one to point at openjdk-7 instead of openjdk-6 – I stuck a modified copy of java-common (which determines the default-jdk) in a PPA:

sudo schroot -c oneiric-java-7-amd64-source
apt-get install python-software-properties
add-apt-repository ppa:james-page/default-jdk-7

Obviously this could just as easily be an upgrade to mysql, postgresql or any other package of your choosing.

Setting up Jenkins

I split this rebuild into ‘main’ and ‘universe’ components and used a mutli-configuration project with axis for ‘package’ (containing the list of packages I wanted to rebuild) and ‘version’ (java-6, java-7); I’ve included the config.xml for one of the projects here.  You can also see the configuration here.

The build is a very simple script:

#!/bin/bash
export HOME=/var/lib/jenkins
rm -Rf *
pull-lp-source ${package}
sbuild -d oneiric-${version} -A -n ${package}*.dsc

That should do the trick; you can then submit the project for build and it will gradually churn through the packages emailing you with any failures.

Jenkins for Ubuntu Oneiric: Call for Testing

Jenkins 1.409.1 has landed in Ubuntu Oneiric Ocelot so now is the time to try it out and help with testing this new distribution model for Jenkins!

Getting Started

If you are not using the development release of Ubuntu, Oneiric Ocelot, grab a copy of the latest alpha-3 milestone pre-release:

• ISO Images: http://cdimages.ubuntu.com/releases/oneiric/alpha-3/
• Cloud Images: http://cloud-images.ubuntu.com/releases/oneiric/alpha-3/

Note that alpha releases are NOT always stable – be careful!  Jenkins is quite happy running on either a Desktop or a Server so take your pick.

Once you have an install of Oneiric Ocelot…

apt-get install jenkins


…should get you going.  Jenkins should startup and be visible on http://localhost:8080 (or the name of your server if you are installing it elsewhere).

A few things to note…

The package for Ubuntu differs in a few ways from the upstream Jenkins .deb packages you might have been using to-date:

• Plugins are not bundled by default – normally Jenkins ships with the following plugins:
  • SSH Slaves Plugin
  • Maven Integration Plugin
  • Subversion Plugin

  Plugins should be compatible with this distribution packaged version of Jenkins – these can be installed through the ‘Manage Plugins’ option within Jenkins as normal.  I would recommend installing the Maven Integration Plugin at a minimum.

• Native OS integration features that are disabled:
  • Solaris: support for libzfs and libembeddedsu4j.
  • Windows: native control of remote windows slaves, native integration with Windows process management, windows service deployment of Jenkins

  It is still possible to use Windows slaves; however they will need to be launched using JNLP from the slave itself.

• Branding: if available on the client, the Web UI will use the Ubuntu font.
• Packages + wrapper scripts are provided for:
  • Monitoring of arbitrary jobs: jenkins-external-job-monitor + see man jenkins-monitor-job for more details.
  • Jenkins CLI: jenkins-cli + see man jenkins-cli for more details.

Using Jenkins

The Jenkins website has a great getting started guide – see https://wiki.jenkins-ci.org/display/JENKINS/Use+Jenkins.

Obviously you can skip the ‘Installing Jenkins’ section.

Reporting Bugs

If you find any problems please report bugs here https://bugs.launchpad.net/ubuntu/+source/jenkins

You can also find me (jamespage) in #ubuntu-server on Freenode IRC most weekdays.

Enjoy…

Avoiding missing-classpath lintian warnings with Maven based packages

Whilst I’ve been preparing the large dependency chain to support Jenkins for upload into Debian and Ubuntu I came across a new lintian warning:

libakuma-java: missing-classpath libjna-java


This is a new check that validates that the jar files associated with a debian -java package have a Class-Path entry set in the META-INF/MANIFEST.mf file.

This is used by wrapper scripts to dynamically set the classpath for Java applications at runtime based on the libraries that they use.

However, if you are packaging Maven based projects you don’t get this set for free (although that would be a great feature for maven-debian-helper).

Luckily the javahelper package has a CDBS class that can help out; simply add javahelper to the Build-Depends for your package then follow these steps:

1) debian/rules

Add the javahelper.mk class into the make file:

#!/usr/bin/make -f

include /usr/share/cdbs/1/rules/debhelper.mk
include /usr/share/cdbs/1/class/javahelper.mk
include /usr/share/cdbs/1/class/maven.mk

...
2) Add a .classpath file for the binary package

In my case this gets dropped into debian/libakuma-java.classpath

usr/share/java/akuma.jar /usr/share/java/jna.jar

Rebuild your package; the jar files should now have Class-Path entries and you should be lintian clean.

Enjoy.

Jenkins for Ubuntu Natty: Call for Testing

Many of you may have wondered what I have been doing since UDS-N; some of you may have know that I have been working on packaging Jenkins (nee Hudson) from source along with all of its build dependencies (all 80 of them).

After just over 3 months I’ve managed to get a 99% built from source Jenkins into the Hudson Ubuntu Testing PPA; this is core Jenkins only at this point in time; no plugins come bundled (although you can still install them using the Plugin Manager).

Note that the PPA only provides packages for the Natty Narwhal development release.

If you want to help out with testing, read on….

Getting started

So first things first; lets add the Jenkins testing PPA and install the jenkins package:

sudo add-apt-repository ppa:hudson-ubuntu/testing
sudo apt-get update
sudo apt-get install jenkins

Wait some time and Jenkins should be up and running.

Accessing Jenkins

Once the package has installed you should be able to access Jenkins on http://localhost:8080 (or whatever server you installed it on).

Jenkins comes completely empty; no jobs, minimal configuration, no security.  Follow the normal Jenkins ‘Use Jenkins‘ guide to get started.

Reporting problems

This is a testing package so it ain’t going to be perfect. Please join the hudson-ubuntu-users team and subscribe to the mailing list on launchpad to discuss general issues and report problems.

You can also find me (jamespage) in #ubuntu-server on Freenode if you like to use IRC.

Whats still todo?

Well as I said this is 99% built from source; a few packages are either binary packages (built from jars) or packages that bundle their dependencies in the upstream source bundle. I’m working with upstream(s) to get these issues sorted.

That said, everything that is installed on your server or desktop in the jenkins package IS built from source. Offending packages are either used for testing or generating documentation during library builds.

I’m also going to start working on packaging a few core plugins from source (see the associated blueprint for details); for the time being you can download and install from within Jenkins. I will also create a patch so that this feature can be disable if required.  Please feel free to apply to join the team if you want to package a plugin.

The Future

Well ultimately probably the main Ubuntu archive and Debian; but lets leave that until next release….

And if you were wondering, Hudson renamed itself to Jenkins in late January (hence the Hudson name keeps creeping into this post).

Trip Report: FOSDEM 2011

With 250+ lectures and 5000 attendees FOSDEM is by far and away the largest software development event that I have attended.

The entire event is organised by the community for the community and takes place at the ULB Campus in Brussels.

I spent most of my weekend in the Free Java Devroom; for a language which has fallen behind and become less popular in recent years, the room was extremely well attended to the point where people where being turned away.  I suspect that this was due to the quality of the schedule and the recent changes in the governance structure of the OpenJDK project generating quite alot of interest in the community.

The Free Java Devroom was run by Tom Marble who did a great job keeping things on track and on-time over the weekend.

OpenJDK

Mark Reinhold (Chief Architect, Java Platform Group, Oracle) kicked off the proceedings with a presentation about the current state of the OpenJDK project.

It was clear that Oracle is 100% behind Java; anything at Oracle that is not a database is written in Java so their business is dependent on the popularity of this platform. They currently employ over 20,000 Java developers!

For those of you who don’t know OpenJDK 7 went through a de-scoping exercise shortly after Oracle acquired Sun Microsystems; this re-planning means that OpenJDK 7 should be with us by July 2011 (instead of the end of 2012) and will include the following new features:

• Project Coin; a number of minor language improvements.  Joe Darcy gave an excellent overview of these features in a separate session.
• InvokeDynamic; vm and language extensions to improve performance of dynamic languages built on Java
• Fork/Join; updates to the concurrency and collections frameworks under JSR-166.

The following features have been dropped from OpenJDK 7 and will be delivered in late 2012 in OpenJDK 8:

• Project Jigsaw; modularization of the Java platform and standardization of dependency declarations ala Debian packaging style. Mark ran another session specifically focused on this project.
• Project Lambda; higher developer productivity and better leveraging of multi-core CPUs.
• Remaining language changes from Project Coin.

I understand the need for the split in order to deliver incremental improvements to the Java platform sooner rather than later; however it is frustrating that Jigsaw won’t be delivered until late 2012.  I think that this project would have moved the Java development community forwards in terms of improving dependency management of Java libraries and applications which will make the lives of Linux distribution packagers much easier.

Mark then went on to talk about the new OpenJDK Governing Board that was recently announced.  This generated a-lot of discussion as the board has been appointed rather than elected; 2 people from Oracle (including Mark), 1 from IBM (effectively killing the Apache Harmony project but thats old news now) and two from the community at large (one who is from Eclipse).  Surprisingly other commercial open source companies such as Red Hat who make a significant contribution to OpenJDK are not represented on the board.   The general feel was that for an open source project this was not the right approach; it was suggested that the board be increased to 7 individuals (the last two being elected from the community) to achieve some balance in the governance of the project.

Oracle’s ownership of the Test Compatibility Kit for Java and the licensing of it to projects is generating a lot of ill feeling in the community and with other companies wishing to build compatible Java runtime environments.  Again this generated a-lot of debate; Azul were present in the room and have been waiting for a TCK for over a year now – supposition is that Oracle is blocking this for commercial reasons.

Java Packaging For Linux Distributions

One hour of the schedule was dedicated to packaging of Java for Linux distributions; unfortunately Thierry Carrez was unable to attend and present about ‘Why Linux Distro’s hate Java’ due to illness.

Torsten Werner (Debian Developer, Java Packaging Team) kicked off with a presentation about the history of Java in Debian and the challenges currently faced by the Java Packaging team. The Java tool-chain in Debian is now looking pretty good; Ant and Maven are both packaged with associated CDBS and maven-*-helper packages to aid building Java from source.

Stanislav Ochotnicky (Co-founder Fedora Java SIG) then gave a presentation on what Java upstream projects can do to help downstream Linux distributions package their applications and libraries more easily. This boiled down to a few key points:

• Don’t patch dependent projects as this is effectively branching; get the feature accepted upstream rather than maintain the branch.
• Avoid including obscure, unmaintained projects in your projects dependency chain.
• If you do bundle dependent libraries with your source code, make it clear what they are and which versions you are using.
• If you are using Maven, make the most of it meta-data that the project definition files can include to describe your project.
• In-fact use Maven; it makes packaging easier

I think that this last point is a key point for Linux distro’s going forwards – although Maven is a much larger packaging challenge in terms of dependencies and complexity when compared to Ant, its by convention approach to laying out a Java project, defining meta-data and defining dependencies should ultimately make Maven based projects much easier to package due through tool-chain automation.

The Best of the Rest

In addition to attending the majority of events in the Free Java Devroom I also managed to catch the keynote from Eben Moglen ‘Why Political Liberty Depends on Software Freedom more than even’. Although I don’t agree with everything he said, he did make some good points about Free and Open Source software solutions challenging traditionally propriety/closed source markets; the concept of using wifi mesh networking to remove reliance on network delivery from centrally controlled companies is a neat idea to provide independence and choice (and resilience when the government disables central internet services…).  He talked quite a bit about the ‘Freedom  Box’; an initiative to deliver a small, cheap inexpensive computer to secure freedom in the home – I noticed that there is already a Debian project aligned to this initiative.

Summary

It is clear that Java is at a critical stage in its lifecycle; the decisions that Oracle makes around how this platform will be developed and governed going forwards will be key.   I think that Mark Reinhold has a very difficult job todo; balancing the commercial demands of Oracle whilst maintaining the support of communities who both develop and use the Java platform will be key.

I also think that FOSDEM is a great conference that I would recommend to anyone that is involved in the world of Free and Open Source Software; thanks to all of the volunteers who make FOSDEM happen!

JOnAS for Ubuntu: Testing

One of the work items that I picked up at UDS-N in Orlando was to produce some basic Ubuntu packaging for JOnAS (Java Open Application Server), a Java Enterprise Edition 5 certified application server stack.

Due to the complexities of packaging this stack from source and with the objective of maintaining JEE 5 certification, this packaging has been produced from the binary distribution of JOnAS 5.1.5; it will allow you to get up and running with JOnAS on the Natty Narwhal development release of Ubuntu server:

Grab the Latest Natty Narwhal Development ISO

wget http://cdimage.ubuntu.com/ubuntu-server/daily/current/natty-server-amd64.iso

So you can either burn a CD and install on some hardware; or as I normally do for testing install in a Virtual Machine.

Enable the JOnAS Ubuntu Packagers Testing PPA

Once you have installed  Natty Narwhal onto the infrastructure of your choice, you will need to enable the PPA (Personal Package Archive) that contains JOnAS for Ubuntu:

sudo apt-get install python-software-properties
sudo add-apt-repository ppa:jonas-ubuntu/testing

This will setup the JOnAS Ubuntu Testing PPA for use on your Ubuntu server.

Install JOnAS for Ubuntu

Now you are ready to install JOnAS 5.1.5:

sudo apt-get update
sudo apt-get install jonas

This may take some time; the package for OpenJDK and JOnAS are both quite large.

And thats it; you should be able to access your JOnAS instance on http://localhost:9000/. You can then follow the JOnAS Getting Started guide; the examples are located in /usr/share/doc/jonas/examples.

Note that this is a first cut testing package so if you find any issues please email jonas-ubuntu at lists dot launchpad dot net.

Enjoy