Improve Server Install and Upgrade Testing

Registered by Robbie Williamson on 2012-04-16

The existing Ubuntu Server ISO Tests (see http://iso.qa.ubuntu.com/) have been broadley automated and have provided a good initial foundation for ensuring the quality of the Ubuntu Server product.

In order to further improve the day-to-day quality of the Ubuntu Server product the depth and breadth of automated and manual testing undertaken during the Q cycle will be improved.

Goals for Q:

Migrate existing tests to UTAH and improve depth and breadth of install and upgrade testing.

Ensure good manual testing coverage at key milestones during development cycle.

Blueprint information

Status:
Complete
Approver:
Dave Walker
Priority:
High
Drafter:
Ubuntu Server Team
Direction:
Needs approval
Assignee:
James Page
Definition:
Approved
Series goal:
Accepted for quantal
Implementation:
Implemented
Milestone target:
milestone icon ubuntu-12.10
Started by
James Page on 2012-06-06
Completed by
James Page on 2012-10-16

Related branches

Sprints

Whiteboard

Summary of Discussion at UDS:

Current test cases are OK - but all need to be migrated to UTAH early during the Q cycle.
Some gaps in upgrade testing - members of the server team to identify suitable test cases so that the QA team can setup more regular upgrade testing.
More requirements identified for UTAH - specifically multi-nic and USB install support.
Still need to ensure that we get good manual testing coverage at key milestones - ubuntu-server list will be used to call for testing.
MAAS test cases need some improvement and automation.

Session notes: http://pad.ubuntu.com/uds-q-servercloud-q-server-iso-tests-review

Assumptions:

- No new tests will be automated using the existing Ubuntu Server ISO Testing Framework.
- All existing tests will be migrated by the Platform QA team as UTAH comes online.
- Existing test format is still valid for UTAH and existing tests can be extended in place.
- The existing upgrade testing framework can support the requirements of the server team.
- OpenStack testing is not covered under the scope of this blueprint.
- Preseed based install testing is still sufficient to prove the ISO install experience.
- Manual testing still has its place a key points in the development cycle.

User Stories: N/A
Test Plan: N/A
Release Note: N/A

(?)

Work Items

Work items for quantal-alpha-1:
[smoser] Call for testing of ISO images - Alpha 1: DONE

Work items for quantal-alpha-2:
[james-page] Call for testing of ISO images - Alpha 2: DONE

Work items for quantal-alpha-3:
Call for testing of ISO images - Alpha 3: DONE

Work items for ubuntu-12.10-beta-1:
Call for testing of ISO images - Beta 1 (missed - sorry): POSTPONED
General call for testing of Ubuntu Server Quantal: DONE

Work items for ubuntu-12.10-beta-2:
[ivoks] Expand existing test cases to support different configuration options for mail-server: POSTPONED
[james-page] Work out better more focussed upgrade testing for server tasks: DONE
[james-page] Call for improvement of test cases and coordination with migration to new harness: POSTPONED
[james-page] Migrate existing ubuntu-server-iso-testing test cases to UTAH: DONE
[james-page] Review server seeds for gaps in testing of packages in main: DONE

Work items for ubuntu-12.10:
Improve MAAS test cases: POSTPONED
[james-page] Plug the iSCSI tests into UTAH: POSTPONED
Ensure UATH can support USB installation as well: DONE
Ensure multi-nic testing can be supported: DONE
Call for testing of ISO images - 12.10 RC: TODO
[pwlars] Add a profile to auto upgrade testing to support VM upgrade testing (nested VMs): POSTPONED
[serge-hallyn] Provide a test to check the status of the VM after upgrade (nested VM's): POSTPONED

Dependency tree

* Blueprints in grey have been implemented.