Standard SRU testing (everything but kernel)
Today, testing a package may be slow because of the waiting period that compensate the lack of rigorous test suites to regression-test changes, lack of hardware for QA’ing, we are mainly relying on community for testing, the amount of patches that get rolled can be important,... All of this can lead to a bug with a known fix going unaddressed in Ubuntu for several months.
We need to shorten the delay of validation for SRUs and being able to provide adequate resolution on an ETA for a fix being included in Ubuntu.
The goal of this session is to discuss and decide how to improve Standard SRU testing (everything but kernel), make it more efficient, increase the coverage of the QRT and build a test harness to run daily.
Blueprint information
- Status:
- Complete
- Approver:
- Pete Graner
- Priority:
- Essential
- Drafter:
- Jean-Baptiste Lallement
- Direction:
- Approved
- Assignee:
- Canonical Platform QA Team
- Definition:
- Approved
- Series goal:
- Accepted for precise
- Implementation:
-
Implemented
- Milestone target:
- None
- Started by
- Gema Gomez
- Completed by
- Gema Gomez
Whiteboard
= Existing testing process overview =
The tasks to be fulfilled are:
1. Review the test case and reproduce the issue
2. build in a clean build environment
3. verify the package still installs
4. verify the package upgrades cleanly
5. verify the package still functions properly
6. verify the bug is fixed using the provided test case
7. verify that it introduces no regression
Today we are relying on the community to achieve theses tasks. The community usually do, to some level, whole or part of tasks 1, 3, 4, 5 and 6, rarely 7.
Step 2 (build) is done automatically by the build environment when the package is published to -proposed.
In any case, we need a human intervention to test the fix and above all that is introduces no regression.
[hggdh2] We should consider expanding test case description: currently it is difficult to quickly find if a test case requires a complex setup or can be automated.
[hggdh2] Another point, brought up on a kernel SRU: if there is a failure on SRU testing, the subject matter experts must provide a reason of why it should still be accepted in the bug's comments; final decision comes from ?. Without the reasoning it will be a failure; the same reasoning can be applied in case of success, but subject matter experts would rather have a failure.
= References =
* General SRU process: https:/
* Kernel SRU: https:/
* Security Updates: https:/
* SRU Status: http://
* Rework the Pending SRU list to display the following information:
* Split the list in 2 parts for each release: Supported package, and packages supported by the community
* Add the importance of the task
* Add an information to display if the description includes a test case or not.
* Indicate the status of verification + regression testing
* Setup a build environment on the QA Lab for each supported release/
* Automate the run of the QRT when a package is published to -proposed
* Automate the run of the package's test suite when a package is published to -proposed
* Build a report to publish the regression test results
* Start counting how many regressions happen for SRUs in the current cycle so that we have metrics for next UDS, which packages do regress (possibly add to people.
* Capture the test cases that would have avoided that regression from happening
Work Items
Work items for precise-alpha-2:
[patrickmwright] Setup a build environment on the QA Lab for each supported release/
[gema] Start counting how many regressions happen for SRUs in the current cycle so that we have metrics for next UDS, which packages do regress (possibly add to people.
[gema] Gather the test cases that would have avoided that regression from happening: POSTPONED
Work items for ubuntu-
[kate.stewart] Send an announcement to ubuntu-devel about the new requirement to add a test script with the SRU whenever applicable: BLOCKED
[jibel] Automate the run of the package's test suite when a package is published to -proposed: POSTPONED
[jibel] Update the pending sru report to show which packages must be tested by canonical and which one must be tested by the community: POSTPONED
[nuclearbob] Automate the run of the QRT when a package is published to -proposed: DONE
[nuclearbob] Build a report to publish the regression test results: POSTPONED
Dependency tree
* Blueprints in grey have been implemented.
