heno log

As promised, here is picture proof of Ania and Beata on the roof of Africa:

And as it happens, they are both Ubuntu users

We are seeking an experienced QA manager with background in open source to lead the Ubuntu QA team at Canonical.

Quality is key to our whole project, but as with many things Ubuntu, QA for the distro is a bit different than QA most other places. In traditional tech companies that deliver a moderate number of products, it is possible to draw up test plans from the original specifications and test strictly against those before releasing.

At Canonical we do perform this type of structured testing for projects that originate here, like Launchpad, Bazaar or the DX components, but for the vast majority of software we ship with Ubuntu this simply isn’t possible. The number of packages in the distrubution is vast and their design can be changed by upstream developers on short notice. This all happens on a 6 month release cycle, which is fast for an operating system, and of course the QA team needs to be working to that schedule as well.

In fact, much of the testing of the Ubuntu platform is currently performed by our early-adopter community, running the Alpha and Beta milestones and reporting bugs. They actually do an impressive job of un-earthing most issues very quickly, but just managing the growing volume of bug reports from that process to ensure we harness its power is a challenge in itself. And unfortunately, that form of testing does have some gaps, esp. for new software or setups that are less common in the community. In this landscape the approach we take to quality assurance must be a hybrid that can draw on the power of the community while continuing to raise formal standards and cover all core components.

We’re looking for someone who can bring new ideas and energy to this task — someone who can bring innovation to quality assurance to match the ongoing innovation that is the Ubuntu project. Are you this person, or do you know someone who might be? Click here to read the posted job description and apply!

And yes, for those who know me and are wondering, what I am describing here  is my own current position I’m moving to the Information Systems Development Team at Canonical (who work on things like the Ubuntu Shop, training portal and various internal systems) to become their Information Architect. I look forward to working on user-friendly and scalable designs for these systems, but first we need a new QA manager!

Jonathan, you are right to call for regressions to be minimised — these bugs have a greater impact on the user experience than most. However, I don’t think delaying the release is something we should  do lightly. Once we start accepting delays in a time-based release schedule we will soon start to see creep where people participating in the release start to figure in the option of having a delay.

In my view, we need to tighten the general quality of Ubuntu, and esp. regressions, while continuing to meet our strict release schedule. We have a series of milestones in the release cycle where we freeze the archive (to an extent) and test. We need to leverage this facility better by applying more testing — and more effective testing — at the early milestones to catch regressions while there is still enough time and wiggle room to fix them.

Yes, there are conceivably some regressions we should block a release on in the future but the first step is to flag and track them properly. If we can detect potential regressions early enough we can avoid them by means other than delays — from proper fixes, to temporary work-arounds or rolling back package versions. But to have these options we must first chart the problems and get better at catching them early.

To this end we recently launched a simple system for tracking regressions in both the development release and stable releases. The information is simply recorded with tags in Launchpad and displayed on a simple page. The data is also used on the more general package status pages. We will add historical views, statistics and graphing over time (and a native Launchpad feature would be great!) but to begin with we felt that it was important to just start recording the information.

We did use the regression tracking system to identify high-impact bugs for intrepid, but it’s in Jaunty that we will really see its full benefit when we can use it through the whole cycle and use the Intrepid numbers as a simple benchmark.

I agree with Mackenzie that we need to test more and test early. Automated testing should indeed form part of the answer as should structured manual testing. You can run a basic set of hardware support tests already with the checkbox test tool. It currently has a selection of interactive and automated tests, but a great deal more can be developed here. Ara is working on a suite of desktop automation tests using the LDTP framework. She has been running these regularly against intrepid milestones and posted the results here. She and Marc are currently working on integration with checkbox to streamline the running of the tests and reporting of results.

We are still just scratching the surface here though and our next big challenge is to extend the test coverage across the desktop, make the tests more detailed and (probably the biggest challenge) keep them up-to-date.

Thanks for putting together a list of applications that need testing scripts! If people can contribute testing stories there we can run them manually to start with and automate them over time. I took the liberty of moving your wiki page over to the Testing section where we already have related material – hope you don’t mind.

Questions or ideas are always welcome on the qa mailing list and in #ubuntu-testing on freenode.

As 8.04 is an LTS release we will be preparing regular updates including updated CD images. Packages targeted for the CDs first go into the hardy-proposed archive where they can be tested by users who have opted in.

During the run up to a regular release we get invaluable feedback from the huge number of people running the development release on a range of hardware. It would be a great help if just a fraction of that group would countinue to help with testing by running hardy-proposed.

To enable hardy-proposed simply tick the Proposed updates box on the Updates tab in Software Sources (System -> Administration -> Software Sources). You should also re-enable Apport so we can collect full crash reports (see instructions). And as always non-crasher bugs are best reported with the Help -> Report a problem menu item in the individual applications.

To register the fact that you are running hardy-proposed, just add a comment to one of these tracking bugs. This makeshift setup will give us an idea of how many people are running hardy-proposed (while we implement a more elegant tracking solution).

So please help us with the 8.04.1 release preparations by testing hardy-proposed and filing bugs!

As you may know, the Ubuntu team is working on a mobile version of the OS for mobile internet devices. But because there isn’t much of this hardware around, the UME builds don’t get the natural community testing that the desktop and server editions do. But if you are interested there is a way you can help!

The mobile environment can run in a Xephyr window. Setting up this environment is fairly easy and is described in detail here.

If you want to help with structured testing please follow the UME test cases starting with the desktop test. And if you find any bugs please file them under the ubuntu-mobile project in Launchpad.

If you have questions you can usually find cgregan, davmor2 or myself (heno) in the #ubuntu-testing channel. Have fun