Adding test coverage to Ubuntu

Registered by Gema Gomez on 2012-04-30

Start building a comprehensive regression testing suite, this blueprint is a discussion to decide where to start adding test cases, so basically choose the most important missing tests and have a clear list of priorities.

Blueprint information

Status:
Complete
Approver:
Pete Graner
Priority:
High
Drafter:
Gema Gomez
Direction:
Approved
Assignee:
Canonical Platform QA Team
Definition:
Approved
Series goal:
Accepted for quantal
Implementation:
Implemented
Milestone target:
milestone icon ubuntu-12.10
Started by
Gema Gomez on 2012-11-13
Completed by
Gema Gomez on 2012-11-13

Related branches

Sprints

Whiteboard

Next steps to improving coverage during Q:
- Migrate existing tests that are run by the QA team to UATH
[hggdh2, psivaa] * Kernel SRU tests (QRT, xfstests)
[hggdh2, psivaa] * ISO Server Testing (smoke testing)
[jibel] * Upgrade testing
[jibel] * Auto package testing
[nuclearbob] * QRT Testing
[mrazik, retoaded] * Autopilot
[nuclearbob] * Bootspeed
[gema] * Bootspeed testing graphical representation design
[nuclearbob, gema] * Implementing the new graphs

- Adding new tests to UATH:
[gema,mrazik, babyface] * Networking testing
[slangasek] * IPV6 testing

07/20/2012, gema: These test cases are more development or package level test cases than testing we can run at system level on the images. I think they are good to be run at development level but do not grant UTAH migration, since they are very specific and low level.

[hggdh2] * 12.10 kernel testing integration with 12.04 (https://wiki.ubuntu.com/QATeam/AutomatedTesting/UpToDateKernel )
[jibel] * LibreOffice tests
[jibel] * jdk tests
[hggdh2, psivaa] * Adding Kernel SRU testing on XEN
[hggdh2, psivaa] * Adding ecryptfs tests to Kernel SRU testing
[nuclearbob] * HDAudio
[josepht] * Power management (cking)
[pitti] * Automatic suspend/resume UI integration tests, VM based (covering indicator menus, screen savers, inhibition by video player, automatic timeout, and low-on-battery)
 * hardware specific suspend/resume: cking has scripts which do this; apw also has a script for this available by default with checkbox: /usr/share/checkbox/scripts/suspend_test
[babyface] * Adding Chinese images to smoke testing
[babyface,jibel] * Add automated UI driven ubiquity test cases (migrate the tests from http://bazaar.launchpad.net/~ubuntu-server-iso-testing-dev/ubuntu-server-iso-testing/trunk/files/head:/templates.desktop/test_cases/ubiquity/ )
[psivaa] * AMD64 and Mac iso testing

- Adding new HW/infrastructure:
[hggdh2] * Fully automating ARM testing (panda boards)
[retoaded, nuclearbob] * Deploying MAAS in the lab and start using it for provisioning machines

- Keeping track of results
[babyface] * Assign responsibilities to keep an eye on regressions
[gema] * Test case guidelines
[gema] * Analyse coverage as a whole and list holes in our coverage

How to add test jobs to jenkins
- Do not use matrix jobs because there is no plugin to count results in that case

Test Cases to be added:

(?)

Work Items

Work items:
[hggdh2] Move Kernel SRU tests QRT, xfstests to UTAH: POSTPONED
[psivaa] Help hggdh2 move kernel SRU tests to UTAH: POSTPONED
[psivaa] Move current smoke testing server to UTAH: DONE
[javier.collado] Move current smoke testing desktop / alternate to UTAH: POSTPONED
[jibel] Move Upgrade testing to UTAH: POSTPONED
[jibel] Move Auto package testing to UTAH: POSTPONED
[nuclearbob] Move QRT Testing to UTAH: POSTPONED
[mrazik] Adapt Autopilot tests to work with UTAH: DONE
[nuclearbob] Move bootspeed testing to UTAH: DONE
[joetalbott] Improve bootspeed testing graphics: POSTPONED
[mrazik] Networking testing, add new tests: POSTPONED
[gema] Assist mrazik in the networking tests addition (no bandwidth): POSTPONED
[stgraber] Send IPV6 testing to the QA team (implemented test, sent to QA): DONE
[gema] Add IPV6 to UTAH: POSTPONED
[hggdh2] 12.10 kernel testing integration with 12.04 (https://wiki.ubuntu.com/QATeam/AutomatedTesting/UpToDateKernel ): DONE
[psivaa] Add Kernel SRU testing on XEN: POSTPONED
[hggdh2] Adding ecryptfs tests to Kernel SRU testing: DONE
[nuclearbob] Add HDAudio tests: POSTPONED
[joetalbott] Add Power management tests from cking: POSTPONED
[pitti] Write screen saver integration tests (starts after configured idle timoeut, wake up by activity, inhibition API, totem inhibition) (tests written, but need rollout to UTAH in jenkins): POSTPONED
[pitti] Write suspend UI integration tests (indicator menus, suspend after configured idle time, suspend after low-on-battery) : POSTPONED
[victor.zhou] Add Chinese images to smoke testing, once tests have been moved to UTAH: POSTPONED
[victor.zhou] Add automated UI driven ubiquity test cases (migrate the tests from http://bazaar.launchpad.net/~ubuntu-server-iso-testing-dev/ubuntu-server-iso-testing/trunk/files/head:/templates.desktop/test_cases/ubiquity/ ): POSTPONED
[psivaa] Add AMD64 and Mac iso testing to the lab: POSTPONED
[hggdh2] Fully automating ARM Kernel testing (panda boards): POSTPONED
[larry-e-works] Deploying MAAS in the lab and make it stable for production: POSTPONED
[nuclearbob] Start using MAAS from UTAH to provision machines: POSTPONED
[victor.zhou] Keep an eye on regressions: DONE
[gema] Write Test case guidelines for UTAH users: DONE
[gema] Analyse coverage as a whole (our coverage is minimal, needs to be improved in all fronts): DONE

Dependency tree

* Blueprints in grey have been implemented.