Cloud Image Testing and Automation

Registered by Ben Howard on 2012-10-03

[RATIONALE]:
Currently the Cloud Images are built and released based on upstream tests. The idea is that if the upstream tests work, that the images will just work. However, due to bugs hit this past cycle, and with the proliferation of new clouds, the need to validate that the cloud images work where they should and characterisiticly are what we say they are, is becoming apparent.

Also, with the proliferation of new cloud vendors, there is a need to automate the release process. Some in the community have asked for a more frequent release cadance for the cloud images.

As a result, for the R-cycle, we will implement testing and automation that will be aimed at ensuring enhanced quality of the images.

[GOAL]:
  1) Implement deeper testing for the Ubuntu Cloud Images
  2) Improve the quality of the Ubuntu Cloud Images by pro-actively looking for regressions and stress testing.
  3) Implement Automatic releases to increase the release cadance for the Cloud Images

The following Tests will be implemented:
* Image Characteristic Tests:
  - Define attributes of a Cloud Image, i.e. prescense of certain files, file formats, and package sets. For example:
        - precences of /etc/cloud and files under it
        - check boot loader configruation for grub and grub2
        - deeper cloud-init testing
        - check SSH configuration

* Bug regression tests: look over the Cloud Image bugs related to the bulding of the images. For example:
    - check that /var/log/{btmp,wtmp,lastlog} exist
    - In order for a cloud image bug to be "fixed released" a test should be written to prevent future regressions.

* Image boot tests
   - Test that QCow images boot on KVM and OpenStack
   - Test that outputed files are valid
   - Validate boot loader and console configurations

* Image proactive bug searching in Cloud instances
   - Fully excersize disk, I/O and network looking for potential bugs
   - Increase Amazon instance sizes tested
   - Excercise -proposed updates to check for issues which may cause problems
   - Use of Juju for testing Juju/Cloud Integration
   - Mock workload tests

* Image performance testing
   - Measure I/O performance to identify problems with kernels or userland configurations caused by updates or build process

* Explore implementation of UTAH testing framework

* Integration and publication of results to Jenkins Instance
   - Add the daily images to iso.qa.ubuntu.com
   - Automatically update test results to iso.qa.ubuntu.com

For Automation:
* Integrate testing into image build process.
   - Daily images for the development release must pass characteristic and regression tests.
   - Daily images for the stable releases must pass all tests to be published
   - Milestones for the development image are subjected to all tests suites
   - Introduce automatic promotion of QA'd daily images when the daily has a new kernel or boot-criticial (i.e. reboot required to use) package.
       - Automate email announcement generation
       - Implement Twitter Announcements
       - RSS Feeds

Feedback Requests (via the two major Ubuntu Cloud Lists)
https://lists.ubuntu.com/archives/ubuntu-cloud/2012-October/000839.html
https://groups.google.com/forum/?fromgroups=#!topic/ec2ubuntu/HLjoxOgOJ10

Blueprint information

Status:
Started
Approver:
Antonio Rosales
Priority:
High
Drafter:
Ubuntu Server Team
Direction:
Approved
Assignee:
None
Definition:
Discussion
Series goal:
Accepted for raring
Implementation:
Started
Milestone target:
milestone icon ubuntu-13.04-beta-1
Started by
Antonio Rosales on 2012-12-12

Related branches

Sprints

Whiteboard

[TEST PLAN]
The core of this blue print is adding tests. Therefore the implementation of this blue print is a test itself.

[ASSUMPTIONS]
It is assumed that this will be a multi-cycle blue print that will undergo refinement. Implementation of some pieces may happen in later cycles.

[RISKS]
The biggest risk for implementation is cost. For some cloud vendors, the ability to implement all tests would represent a significant financal burden. This risk will be mitigated by stepping tests such that only relavent tests will be done as approprate.

[RELEASE NOTE]
Starting with "r", the Ubuntu Cloud Images are now on a release cadance that follows the kernel SRU process. In general, new Cloud Images for all supported stable releases and long-term releases will follow a few days after availability of kernel, gcc and other boot-critical packages. Further, all released daily builds undergo exhaustive testing to ensure quality and to check regressions, performance and integration into Cloud environments. Users who are interested in tracking the testing can find the test results on http://jenkins.qa.ubuntu.com and http://iso.qa.ubuntu.com.

[FEEDBACK from RFC]:

Suggested for Release Notes and Notification:

1/ release notes for each releases are published and kept on
cloud-images.ubuntu.com. Maybe in /<suite>/release-notes/ with a link
from the individual /<suite>/<version>/ directory? As you remove
versions you still keep the release notes.

2/ Set up a notification method via twitter, or RSS Feed.

3/ Keep sending emails for the time being

(?)

Work Items

Work items:
[utlemming] Implent QA workflow into new build system: DONE
[utlemming] Deterministic testing of QA results: DONE
[utlemming] Restructure build workflow to support automated releases: DONE
[utlemming] Implement RSS feeds: DONE
[utlemming] Implement release notes: DONE
[utlemming] Implement archive job to move old builds to S3: DONE
[utlemming] Migrate 10.04 to new workflow: DONE
[utlemming] Migrate 11.10 to new workflow: DONE
[utlemming] Migrate 12.04 LTS to new workflow: DONE
[utlemming] Migrate 12.10 to new workflow: DONE
[utlemming] Migrate 13.04 to new workflow: DONE
[utlemming] Implement deep image characteristic tests: INPROGRESS
[utlemming] Implement performance testing: INPROGRESS

Dependency tree

* Blueprints in grey have been implemented.