Improve community kernel SRU testing

Registered by Ara Pulido on 2011-09-27

SRU testing is about validating that package updates don't introduce problems on stable releases. The testing includes regression testing performed by the QA team, security testing by the Security team, certification testing by the Certification team and verification by the Kernel team. There is also some testing performed by the Ubuntu community coordinated by the QA team, but this is mostly achieved on a best effort basis. The purpose of this blueprint is to increase community participation and include their successful testing rather than just bug reports.

The current process for getting community participation is by commenting on the bugs fixed by the SRU when the package is building for the proposed archive. The comment describes how to enable this archive where the package will be ready in a few hours and then asks the subscribers to please test with feedback. Another process specific for kernel packages is for development releases to be announced on voices.canonical.com and proposed kernels may also be announced eventually. As can be seen from the pending SRU reports, this has not resulted in much participation and the responsibility then falls upon the QA team.

The first objective is to increase community participation for proposed kernels. The most significant value is to increase the probability that verification testing is performed by someone in the community within a reasonable delay. As a side effect, another value is to relieve the QA team from some testing which can otherwise result in additional delays. The SRU release team would benefit from having more people testing with minimal delays to push a new kernel from proposed to updates.

The second objective is to include successful testing from the community rather than just bug reports. This will enable the SRU release team to also consider the number of people having tested the proposed kernels in addition to their existing reports. This should result in greater confidence before pushing a new kernel from proposed to updates.

This is the story to achieve both these objectives:
1. A kernel is added to the proposed archive, so the user is notified that a new kernel is available;
2. The user opens Update Manager where she is informed that the kernel can be tested after rebooting;
3. After booting for the first time with a proposed kernel, Checkbox prompts the user to run the SRU suite;
4. If a test fails, Apport is invoked and tags the bug with "regression-proposed" for the current reports;
5. After testing, all test results are submitted to Launchpad where additional reports can be generated.

Blueprint information

Status:
Not started
Approver:
Ara Pulido
Priority:
Essential
Drafter:
Marc Tardif
Direction:
Needs approval
Assignee:
None
Definition:
Approved
Series goal:
None
Implementation:
Deferred
Milestone target:
None

Related branches

Sprints

Whiteboard

== Definition of done ==

* Update Manager informs the user that they will be prompted for testing when rebooting into a proposed kernel.
* Checkbox starts automatically when a new session is opened the first time with a proposed kernel.
* Checkbox prompts the user for automated and maybe some manual tests from the SRU suite.
* Reports are generated from Launchpad to summarize community testing of the proposed kernel.

== Who uses Proposed ==

There are two kinds of people who use Proposed:
1. People who are having a problem, and who are told that a package in Proposed may fix the problem.
  - Question: What kinds of people? Would this ever include support customers, for example?
2. People who test proposed updates because they like contributing to QA.
  - We have no way of knowing how many people this is.

Question:
- Could Proposed become a NotAutomatic pocket, like backports is? So that people can test one proposed package without getting all the others.
  - People in group #2 above would still want to know immediately about any proposed update. Does that mean it shouldn't be NotAutomatic?

Work items:
[mpt] Work with cr3 to redesign how Proposed updates are configured and announced, based on the answers above.

(?)

Work Items