Improve Server Install and Upgrade Testing

Registered by Robbie Williamson

The existing Ubuntu Server ISO Tests (see have been broadley automated and have provided a good initial foundation for ensuring the quality of the Ubuntu Server product.

In order to further improve the day-to-day quality of the Ubuntu Server product the depth and breadth of automated and manual testing undertaken during the Q cycle will be improved.

Goals for Q:

Migrate existing tests to UTAH and improve depth and breadth of install and upgrade testing.

Ensure good manual testing coverage at key milestones during development cycle.

Blueprint information

Dave Walker
Ubuntu Server
Needs approval
James Page
Series goal:
Accepted for quantal
Milestone target:
milestone icon ubuntu-12.10
Started by
James Page
Completed by
James Page

Related branches



Summary of Discussion at UDS:

Current test cases are OK - but all need to be migrated to UTAH early during the Q cycle.
Some gaps in upgrade testing - members of the server team to identify suitable test cases so that the QA team can setup more regular upgrade testing.
More requirements identified for UTAH - specifically multi-nic and USB install support.
Still need to ensure that we get good manual testing coverage at key milestones - ubuntu-server list will be used to call for testing.
MAAS test cases need some improvement and automation.

Session notes:


- No new tests will be automated using the existing Ubuntu Server ISO Testing Framework.
- All existing tests will be migrated by the Platform QA team as UTAH comes online.
- Existing test format is still valid for UTAH and existing tests can be extended in place.
- The existing upgrade testing framework can support the requirements of the server team.
- OpenStack testing is not covered under the scope of this blueprint.
- Preseed based install testing is still sufficient to prove the ISO install experience.
- Manual testing still has its place a key points in the development cycle.

User Stories: N/A
Test Plan: N/A
Release Note: N/A


Work Items

Work items for quantal-alpha-1:
[smoser] Call for testing of ISO images - Alpha 1: DONE

Work items for quantal-alpha-2:
[james-page] Call for testing of ISO images - Alpha 2: DONE

Work items for quantal-alpha-3:
Call for testing of ISO images - Alpha 3: DONE

Work items for ubuntu-12.10-beta-1:
Call for testing of ISO images - Beta 1 (missed - sorry): POSTPONED
General call for testing of Ubuntu Server Quantal: DONE

Work items for ubuntu-12.10-beta-2:
[ivoks] Expand existing test cases to support different configuration options for mail-server: POSTPONED
[james-page] Work out better more focussed upgrade testing for server tasks: DONE
[james-page] Call for improvement of test cases and coordination with migration to new harness: POSTPONED
[james-page] Migrate existing ubuntu-server-iso-testing test cases to UTAH: DONE
[james-page] Review server seeds for gaps in testing of packages in main: DONE

Work items for ubuntu-12.10:
Improve MAAS test cases: POSTPONED
[james-page] Plug the iSCSI tests into UTAH: POSTPONED
Ensure UATH can support USB installation as well: DONE
Ensure multi-nic testing can be supported: DONE
Call for testing of ISO images - 12.10 RC: TODO
[pwlars] Add a profile to auto upgrade testing to support VM upgrade testing (nested VMs): POSTPONED
[serge-hallyn] Provide a test to check the status of the VM after upgrade (nested VM's): POSTPONED

Dependency tree

* Blueprints in grey have been implemented.