Upgrade testing for Saucy

Registered by Paul Larson on 2013-05-13

Upgrade testing is currently performed using the https://launchpad.net/auto-upgrade-testing tool. We would like to consolidate everything under UTAH where it makes sense to do so. The UTAH team is currently adding some support for things like image snapshotting and lxc that should make it possible to do upgrade testing under utah as well.

Blueprint information

Status:
Complete
Approver:
Gema Gomez
Priority:
High
Drafter:
Paul Larson
Direction:
Approved
Assignee:
Canonical Platform QA Team
Definition:
Obsolete
Series goal:
None
Implementation:
Unknown
Milestone target:
milestone icon ubuntu-13.10
Completed by
Gema Gomez on 2013-11-07

Related branches

Sprints

Whiteboard

# This can be done by hand for trying the process the first time, but we need should create a script to do this, so that rolling over to a new release is easier, and possibly a wrapper that looks for the images in a predictable location and creates them if they are not there:
1) just virt-manager by hand to create an image for you from an ISO,
give a nice memorable name, like qa-12.04-upgrade
[jibel] The creation is already automated so that would be a regression to create VMs manually. I see the following options to create a base VM
- Fix vm-builder
- or a cloud image to provision the base image like adt or http://ubuntu-smoser.blogspot.fr/2013/02/using-ubuntu-cloud-images-without-cloud.html
2) in that image make sure the root user gets the jenkins/utah pub-key
added to its authorized keys
3) make sure the image has sshd and gdebi installed
4) make a copy of this

Run UTAH using the normal "run_utah_tests.py" script but add these two parameters:
  --skip-provisioning
  --name <name from step 1 above>

# re-use it
copy your original image file over to where libvirt expects it. (I think
its /var/lib/libvirt/images/....)

Existing tests can probably be moved over fairly directly, except one will need to be added that performs the upgrade, and uses "reboot: always" in the control file

--------------------
Session notes
--------------------

* Upgrade test modes are: server, desktop, main packages, universe packages
  - Universe isn't everything in universe, but does include everything with a desktop file in app-install-data
* VMs good enough for now, but are slow. LXC preferred for main and universe upgrade testing when utah supports it - keep VM upgrading for server/desktop so we make sure to test bootloader and kernel upgrades completely
* Utah can be used in one mode to create base images, and another to run the upgrade and post-upgrade tests
* Scripts needed for installing packages for main/universe testing, setting up user/system customization. These are found in auto-upgrade-testing:/share/scripts and are free of dependencies - should be easy to bring across

Plan for tests:
* Bring across existing tests - lxc backend should have something to help deal with changes needed to kernel test for running kernel
* Check for .crash files, but only ones that happen after performing the upgrade (clear /var/crash right after upgrade completes, before rebooting)
* report (but not fail) on packages that are no longer in ubuntu, but are still installed after the upgrade (mostly useful for libs)
* wishlist: upgrade using the UI upgrader
 - basically this would just need to replace the "upgrade" test in desktop with an autopilot test to handle the gui upgrade process
* create a testplan for upgrade testing, start making progress toward additional tests

Reporting:
* would like these to be associated with smoke test results for the day, something that can show people whether it's safe to upgrade on any given day
 - maybe part of the key performance indicators?
* Make the upgrade reporting view link to reports for things like conffiles, and still-installed packages that are no longer in the archive after upgrade

(?)

Work Items

Work items:
create new image creation script for LXC once it's supported (desktop/server only): TODO
New test for .crash files and clear pre-reboot crash files in upgrade test: TODO
New test for packages that are no longer in ubuntu, but still installed: TODO
New test for upgrading using the UI upgrader: TODO
Review tests and plan for future upgrade tests needed: TODO
Add reporting to dashboard for upgrade testing: TODO

Work items for ubuntu-13.06:
[doanac] Create a new jenkins.sh type script that can run tests and provision(if needed): DONE
[doanac] turn jenkins.sh into a templated script that can create jenkins jobs: DONE

Work items for ubuntu-13.07:
[psivaa] create a runlist that does the equivalent of auto-upgrade-testing: INPROGRESS
[psivaa] Hook setup steps needed by some tests in so that they run before the upgrade: TODO
[jibel] Deploy upgrade tests for S and remove obsolete tests: DONE

This blueprint contains Public information 
Everyone can see this information.