QA move to Image Based Updates and click packages

Registered by Gema Gomez

QA move of testing to the new Image Base Updates

Blueprint information

Not started
Needs approval
Canonical CI Engineering
Series goal:
Accepted for saucy
Milestone target:
milestone icon ubuntu-13.10


New OS updates imply two new things QA-wise:
* testing new OS updates
* adapting QA tests to new OS update (read-only system fs)

Also simultaneously switching over to Click packages as the system is r/o.
Testing new OS updates
3 main tests to add to smoke tests:
* bootstrap
* delta updates:
  * with one delta
  * with two deltas
* full update (uses a full image instead of a delta to update)

Updating goes through a series of reboots, so QA testing will need to detect these reboots

There is a DBus service to trigger the OS updates (that settings app hooks into) that the tests should use too to trigger updates

Adapting QA systems
apt-get install not working (read-only fs); problematic for some tests and to pull common deps such as autopilot

(fginther) adapt upstream merger infrastructure to use Click instead of debian, run autopilot tests on device, trigger merge
- Building (sergio)
  - click chroot builder
    - jenkins plugin similar to pbuilder?
  - licence check hook
  - jenkins job which creates ubuntu touch image
    - create a daily image
    - perform an apt-get update/dist-upgrade prior to building ss
  - build with and without coverage
    - execute unit tests (with coverage)
  - create click package
  - verify symbols don’t link to anything outside of the base image or the click package itself.
  - export click package as a jenkins artifact
- Testing (ginther)
  - Testing through click image
  - Install archived click package on top of provisioned device
  - Pass MP path as an argument to source tests
  - Remove click package to restore environment

Need to adapt autopilot tests to launch Click apps rather than directly running binaries

Provisioning device with tests when using click packages (sergiusens)

- Embed the bzr branch somewhere in the click package that we can retrieve.
- Create a set of base emulator dependencies (unity, osk)
- Fetch the click package branch
- Fetch the emulator dependencies
- Bundle the emulators and tests into a python package
- Push to device
- autopilot run [packagename].* (or whatever)

We agreed to use the click application name and parse the installed manifest for the user (click list --manifest) autopilot will have a 'launch_click_app(app_name)' simil interface to launch apps and will hook up to the users installed click list to retrieve the version to generate a full app_id


Work Items

Work items:
Do real smoke testing after performing an update to validate things really work: TODO
Adapt UTAH to test images without using developer-mode: TODO

Work items for ubuntu-13.07:
[sergiusens] phablet-flash flag to pull older images than the latest one: DONE
[sergiusens] phablet-flash flag to be able to pull new images as well as current: DONE

Work items for ubuntu-13.08:
[doanac] Adapt UTAH to handle new images (use developer mode): DONE
[cjohnston] Setup smoke jobs of current apps to use new images: DONE
[javier.collado] Test new OS updates -1 and -2 image delta: INPROGRESS
[pwlars] Agree with doanac in the name of these images and create jobs that make them appear on the dashboard under a new name, for a while there will be results for existing images and new images: DONE
[pwlars] Modify scripts and generate jobs for new images: DONE
[pwlars] Assess the situation with current HW and decide if we need to run new tests on different devices to avoid disruption of old images: DONE
[larry-e-works] Order new hardware if required and attach to phoenix: INPROGRESS
[fginther] Adapt upstream merger infrastructure to use click instead of debian: TODO
[doanac] Launch autopilot tests via a upstart: INPROGRESS
[thomir-deactivatedaccount] Ensute autopilot can be installed on the new images and can run from a click package: TODO
[om26er] Do manual run of autopilot tests on the new images to make sure all pieces are in place (pre-requisite to starting any kind of automation, green light from omer): TODO
[pwlars] Monitor new results for a while until they are stable: INPROGRESS
[barry] Add flag to force full OS update (LP: #1192575): DONE
[stgraber] Will need a new daily-proposed channel on system-image.u.c: TODO
[sergiusens] Seed autopilot in OS images: DONE
[sergiusens] Jenkins click builder plugin: TODO
[sergiusens] Create tool to provision tests for click packages: DONE

Work items for ubuntu-13.09:
[javier.collado] Add new test cases for images updates: TODO
[pwlars] Start running new test cases for updates on regular cadence: TODO
[pwlars] Change dashboard to use new touch_ro version numbers: DONE
Avoid running tests when images are garbage collected: TODO
[doanac] Add support for daily-proposed channel in CI infrastructure: DONE

Dependency tree

* Blueprints in grey have been implemented.