Jonathan, you are right to call for regressions to be minimised — these bugs have a greater impact on the user experience than most. However, I don’t think delaying the release is something we should do lightly. Once we start accepting delays in a time-based release schedule we will soon start to see creep where people participating in the release start to figure in the option of having a delay.
In my view, we need to tighten the general quality of Ubuntu, and esp. regressions, while continuing to meet our strict release schedule. We have a series of milestones in the release cycle where we freeze the archive (to an extent) and test. We need to leverage this facility better by applying more testing — and more effective testing — at the early milestones to catch regressions while there is still enough time and wiggle room to fix them.
Yes, there are conceivably some regressions we should block a release on in the future but the first step is to flag and track them properly. If we can detect potential regressions early enough we can avoid them by means other than delays — from proper fixes, to temporary work-arounds or rolling back package versions. But to have these options we must first chart the problems and get better at catching them early.
To this end we recently launched a simple system for tracking regressions in both the development release and stable releases. The information is simply recorded with tags in Launchpad and displayed on a simple page. The data is also used on the more general package status pages. We will add historical views, statistics and graphing over time (and a native Launchpad feature would be great!) but to begin with we felt that it was important to just start recording the information.
We did use the regression tracking system to identify high-impact bugs for intrepid, but it’s in Jaunty that we will really see its full benefit when we can use it through the whole cycle and use the Intrepid numbers as a simple benchmark.
I agree with Mackenzie that we need to test more and test early. Automated testing should indeed form part of the answer as should structured manual testing. You can run a basic set of hardware support tests already with the checkbox test tool. It currently has a selection of interactive and automated tests, but a great deal more can be developed here. Ara is working on a suite of desktop automation tests using the LDTP framework. She has been running these regularly against intrepid milestones and posted the results here. She and Marc are currently working on integration with checkbox to streamline the running of the tests and reporting of results.
We are still just scratching the surface here though and our next big challenge is to extend the test coverage across the desktop, make the tests more detailed and (probably the biggest challenge) keep them up-to-date.
Thanks for putting together a list of applications that need testing scripts! If people can contribute testing stories there we can run them manually to start with and automate them over time. I took the liberty of moving your wiki page over to the Testing section where we already have related material – hope you don’t mind.
Questions or ideas are always welcome on the qa mailing list and in #ubuntu-testing on freenode.