Automated ubiquity testing with autopilot

Registered by Nicholas Skaggs

Our goal for this cycle is to have daily images tested with the ubiquity autopilot test suite for all desktop manual test entries for each flavor. Let's talk about the current progress and what needs finished.

https://wiki.ubuntu.com/QATeam/AutomatedTesting/ubiquity

Can we blocked merges to proposed based on test results?

Current tests: https://jenkins.qa.ubuntu.com/job/autopilot_ubiquity-devel/

- Push server automated tests to qatracker
- Migrate manual tests to 'optional' once automated by ubiquity testing
- Explore ideas for submitting 'generic' manual test results (ie smoketests) to tracker to be counted
- Create Long term plan for manual testcases that are no longer needed -- how to maintain or not?

Blueprint information

Status:
Not started
Approver:
Jono Bacon
Priority:
Undefined
Drafter:
Nicholas Skaggs
Direction:
Approved
Assignee:
Nicholas Skaggs
Definition:
New
Series goal:
Accepted for trusty
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

current status:
    4 tests running everyday
    testing 6 different flavors of ubuntu

    need to have long period of stable results before we plug into release process
        could block on move from pending/current
        could report to qatracker

tests status
    4 basic tests
        all good
        custom install will see some additional tweaks

    test migration plans
        run all tests in same enviroment
        only have CI pull the "vetted" tests, so we can add new tests as we write them
        coordinate with CI team on best practicies for migration
        ~2-3 weeks vetting before migration
            perhaps seek to push new tests with each milestone

    new tests
        post-install tests
            run test-suite from preseed (smoke tests)
            run after every testsuite

    do we still need manual tests? Assumption that auto testing works with other flavours.
        certain things only a user will see
        UI artifacts are common in the installer
        yes, and perhaps migrate them to run-once.. We would still want user confirmation for tests (though we'd feel better about only having one response of it working)

    working tests:
        Install (entire disk)
        Install (manual partitioning)
        Install (entire disk with lvm and encryption)
        Install (entire disk with lvm)
        english and non-english(random language)
    add for alpha 1:
         Install (auto-resize) - Moved to alpha 2
    add for alpha 2:
         Install (auto-resize)
         Install (erase and re-install)
             reusing home
             upgrading
         expanded manual partitioning
             Install (manual partitioning re-using home partition)
         post installation tests added
    add later:
        boottime options
            oem install
            free software only install
        live session

Auto-Resize Notes:
* requires an 80G disk image size anything less than that ubiquity doesn't provide the option to install alongside.

(?)

Work Items

Work items for ubuntu-14.04:
Review results of automated tests and ensure they are stable no false positive and no false negative and fix or drop flaky tests Plan to do this review on or around 13 Dec in time for alpha 1. : TODO
[jibel] After the review period, hand over project to Ubuntu CI team : INPROGRESS
[nskaggs] Coordinate with cjwatson and plars to integrate this testing with image publication from pending -> current : INPROGRESS
[nskaggs] Coordinate with CI team on best practicies for migrating new tests as we write them : TODO
[jibel] Integrate publication of results from jenkins jobs with ISO Tracker : TODO
[jibel] Add support for reboot after installation and run post-installation tests : TODO
[jibel] Add support to the runner for multi disks and pre-installed disks : TODO
[nskaggs] add tests timeline to the ubiquity wiki page : TODO
[nskaggs] add bugs for needed tests, filed against ubiquity, tag with needs-autopilot-test : TODO
[dpniel] make sure doc/video/tutorial is added so folks can contribute new ubiquity tests successfully : INPROGRESS
[dpniel] finish working tests : DONE
[dpniel] plan what's required to reach target for alpha 2 and add to the wiki page for others to get involved with. : TODO

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.