Automated tests backlog for Utopic

Registered by Jean-Baptiste Lallement

The goal of this blueprint is to track tests failing in CI during automated testing of the phone.

Blueprint information

Julien Funk
Jean-Baptiste Lallement
Canonical Platform QA Team
Series goal:
Accepted for utopic
Informational Informational
Milestone target:
milestone icon ubuntu-14.10
Started by
Leo Arias
Completed by
Leo Arias



We will continue improving the existing automation from

The bugs will be tagged qa-broken-test qa-daily-testing:
In order to highlight bugs to the landing team, tag them with qa-landing-email:

Observations and recommendation

- Things usually break when major things land (unity8, uitk, qt, mir)
- Tests break even when they should have caused CI failures to happen (and thus block the MP)
 - When tests are known to be flaky, teams might ignore failures - messaging-app failure (change to name of icon)
- Many projects have cumbersome release processes (necessitated by the nature of the project, e.g. Unity8, Mir, UITK). This means that landings are fraught with risk and getting fixes in is very slow.
- Some projects don't have adequate test infrastructure (core-apps don't run tests on ARM)
- Test and release cycle is not tight enough. A silo might be tested on 172, land in 174, but be broken by a change in 173.
- Click apps can use Python modules that in turn requires deb packages to work - see
- Landings can frequently contain a lot of code making it difficult to assess the impact


- Use autopackagetest where test suites are stable
- Have more frequent releases (one image per landing)
- Put more importance on having stable, reliable CI results
- Make sure all projects run tests on device or emulator
- Make sure CI tests are passing on branches that are landing - hard rule
- Make sure all CI results are for the very latest image prior to publishing
- Enable large projects to run full set of suites quickly - parallelise
- Have a way to process test runs and possibly rerun failing tests that may have a chance of passing when rerun. Make sure to adequately report the reruns so they can be addressed.
- Test Click apps in CI as they are on the dashboard
- It should be possible to isolate fixes into seperate landings


Work Items

This blueprint contains Public information 
Everyone can see this information.