Improving Smoke Testing Coverage

Registered by Gema Gomez

Have a comprehensive smoke testing suite that is reliable by the end of the cycle

Blueprint information

Status:
Complete
Approver:
Pete Graner
Priority:
Essential
Drafter:
Gema Gomez
Direction:
Approved
Assignee:
Canonical Platform QA Team
Definition:
Approved
Series goal:
Accepted for quantal
Implementation:
Implemented
Milestone target:
milestone icon ubuntu-12.10
Started by
Gema Gomez
Completed by
Gema Gomez

Related branches

Sprints

Whiteboard

Postponed tasks from last cycle:

See https://blueprints.launchpad.net/ubuntu/+spec/other-p-builds-smoke-testing for more details
Strategy to tackle this problem:
- Proposal 1: Go through all the packages that are on an iso from main and add at least one or two test cases for each, positive testing.
- Proposal 2: Go through a list of applications (or groups of packages that functionally make sense to use together) and add one or two test cases for each, positive testing.

indicators, gedit, firefox, libre office,

(?)

Work Items

Work items:
[psivaa] ISO static validation, write a script that validate the ISO and integrate it into UTAH: DONE
[gema] Compile a list of applications that are installed by default by the ISO installers for Desktop and propose two or three basic test cases for each that could be run post install giving us basic confidence that the ISO is good for further testing (to be done after UTAH is in production): POSTPONED
[jibel] Add a test scenario to the automated testing in Jenkins to address full updated installs with Ubiquity (problems to be detected with this testing: bug 743359 and bug 897680), test installers with updates on so that we can catch regressions for alpha3, beta1 and beta2 images (to be added after UTAH migration is complete): POSTPONED
[gema] Get a list of applications from the release manifest and prioritize the test cases (To be done next cycle, when we actually get to add new test cases): POSTPONED
[canonical-platform-qa] Add the test cases resulting from the previous action to smoke testing during Q: POSTPONED
[jibel] Add test for ubuntu-core i386 and amd64: DONE
[jibel] Add some xml output to the current ubuntu-core test so that jenkins can understand them: DONE
[jibel] Add test case documentation (README?) to the ubuntu-core test so that the move to utah is seamless: DONE
[jibel] Upgrade auto-upgrade-tester to latest code base following port to python3 - O->P and P -> Q: DONE
[jibel] Upgrade auto-upgrade-tester to latest code base following port to python3 - L->P: DONE
[psivaa] Add a post-installation test that checks for obsolete content in an image. After installation, update the package cache and count the number of updates available. The test succeeds if the number of updates is less than 10 it fails otherwise (to be done after the utah migration): POSTPONED

Dependency tree

* Blueprints in grey have been implemented.