Staging ISOs before daily smoke testing

Registered by Dimitri John Ledkov on 2012-09-18

Trying to improve ISO testing by running some automated testing before image publication.

Blueprint information

Status:
Not started
Approver:
Pete Graner
Priority:
High
Drafter:
Gema Gomez
Direction:
Needs approval
Assignee:
Canonical Ubuntu QA Team
Definition:
New
Series goal:
Accepted for raring
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

During the quantal cycle there have been times where the daily ISO images had bug or problems on them, which hit many users and or automated jenkins testing.
Ideally we want to prevent users from downloading borked images and hitting/reporting bugs over and over again.
We should discuss and possibly implement something to help with this, e.g.:
* consider uploading images to a staging area, running jenkins test and if jenkin tests pass then publish the iso images
* or for example push jenkin test failure notes to the cdimage / iso-tracker (like the oversize warning)
* think about potentially yanking borked images from cdimage.u.c
* communicate "why there was no daily build yet?", e.g. by pushing "FTBFS" notification to iso-tracker & cdimage.u.c
* communicate respin process in a more automatic way, e.g. on the iso tracker / cdimage.u.c

Objective:
- Try to run some automated testing before the images are published
- Work with Unity testing team to make sure unity works (hook unity jenkins to be dowstream from the desktop-default jenkins test)

(?)

Work Items

Work items:
[cjwatson] Pre-publish images to a staging area, and work with gema on how to trigger testing: DONE
[cjwatson] Pull jenkins results on above smoke-test and then publish (or not) based on results: INPROGRESS
[cjwatson] Keep the image that doesn't pass $somewhere, so we can debug what broke: DONE
[pwlars] Discuss potential triggering methods with Colin for the initial publish/no-publish call: DONE
[pwlars] Get firewall rules changed to allow ssh to cdimage from a QA system to request the images get copied from pending->current: DONE
[pwlars] Modify image sync scripts to sync from pending instead of current: DONE
[pwlars] Setup a new parameterized job to handle requesting the pending->current transition when smoke tests pass, fix job hierarchy to trigger this at the proper place: DONE
[pwlars] modify smoke job creation scripts to include the new pieces to call mark-current, use them to create the new test jobs for saucy: DONE
[pwlars] fix download-latest-test-iso script and modify precise jobs to use it properly: INPROGRESS