Automated ubiquity testing with autopilot

Registered by Nicholas Skaggs

Our goal for this cycle is to have daily images tested with the ubiquity autopilot test suite for all desktop manual test entries for each flavor. Let's talk about the current progress and what needs finished.

Can we blocked merges to proposed based on test results?

Current tests:

- Push server automated tests to qatracker
- Migrate manual tests to 'optional' once automated by ubiquity testing
- Explore ideas for submitting 'generic' manual test results (ie smoketests) to tracker to be counted
- Create Long term plan for manual testcases that are no longer needed -- how to maintain or not?

Blueprint information

Not started
Jono Bacon
Nicholas Skaggs
Nicholas Skaggs
Series goal:
Accepted for trusty
Milestone target:

Related branches



current status:
    4 tests running everyday
    testing 6 different flavors of ubuntu

    need to have long period of stable results before we plug into release process
        could block on move from pending/current
        could report to qatracker

tests status
    4 basic tests
        all good
        custom install will see some additional tweaks

    test migration plans
        run all tests in same enviroment
        only have CI pull the "vetted" tests, so we can add new tests as we write them
        coordinate with CI team on best practicies for migration
        ~2-3 weeks vetting before migration
            perhaps seek to push new tests with each milestone

    new tests
        post-install tests
            run test-suite from preseed (smoke tests)
            run after every testsuite

    do we still need manual tests? Assumption that auto testing works with other flavours.
        certain things only a user will see
        UI artifacts are common in the installer
        yes, and perhaps migrate them to run-once.. We would still want user confirmation for tests (though we'd feel better about only having one response of it working)

    working tests:
        Install (entire disk)
        Install (manual partitioning)
        Install (entire disk with lvm and encryption)
        Install (entire disk with lvm)
        english and non-english(random language)
    add for alpha 1:
         Install (auto-resize) - Moved to alpha 2
    add for alpha 2:
         Install (auto-resize)
         Install (erase and re-install)
             reusing home
         expanded manual partitioning
             Install (manual partitioning re-using home partition)
         post installation tests added
    add later:
        boottime options
            oem install
            free software only install
        live session

Auto-Resize Notes:
* requires an 80G disk image size anything less than that ubiquity doesn't provide the option to install alongside.


Work Items

Work items for ubuntu-14.04:
Review results of automated tests and ensure they are stable no false positive and no false negative and fix or drop flaky tests Plan to do this review on or around 13 Dec in time for alpha 1. : TODO
[jibel] After the review period, hand over project to Ubuntu CI team : INPROGRESS
[nskaggs] Coordinate with cjwatson and plars to integrate this testing with image publication from pending -> current : INPROGRESS
[nskaggs] Coordinate with CI team on best practicies for migrating new tests as we write them : TODO
[jibel] Integrate publication of results from jenkins jobs with ISO Tracker : TODO
[jibel] Add support for reboot after installation and run post-installation tests : TODO
[jibel] Add support to the runner for multi disks and pre-installed disks : TODO
[nskaggs] add tests timeline to the ubiquity wiki page : TODO
[nskaggs] add bugs for needed tests, filed against ubiquity, tag with needs-autopilot-test : TODO
[dpniel] make sure doc/video/tutorial is added so folks can contribute new ubiquity tests successfully : INPROGRESS
[dpniel] finish working tests : DONE
[dpniel] plan what's required to reach target for alpha 2 and add to the wiki page for others to get involved with. : TODO

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.