Ubuntu LEB and LAVA: Current status and future planning for proper image testing and validation

Registered by Ricardo Salveti on 2012-01-31

This session will present how the Ubuntu LEB images are currently tested with LAVA, and discuss about future planning/development to reach the goal of having the image testing mostly automated (and covering how manual test report should also be handled).

Topics:
 * Daily images
 * Snapshots -> LAVA
 * State of the test cases
 * How to handle manual test reports

Blueprint information

Status:
Not started
Approver:
Ricardo Salveti
Priority:
Undefined
Drafter:
Ricardo Salveti
Direction:
Needs approval
Assignee:
Ricardo Salveti
Definition:
Discussion
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

Session Notes:

Ubuntu LEB and LAVA

Topics:
 * Daily images
  - Currently they are "cron jobs" that always gets the latest rootfs and hwpacks and pushes the test runs
  - Problems:
    - Can easily retest the same thing over the time
    - Hard to track the test results per images
    - Missing builds, like respins per day
    - Test list, how it's properly maintained
    - Not necessarily all hwpacks are tested, not even sure if we're testing what we care about
  - Solution:
    - Remove the responsability of the test runs from the validation team and move to the dev platform
    - Jobs at jenkins to create the hwpack and the rootfs images with live-build (cross bootstrap)
    - Hook the images with LAVA, to submit the test requests and present a way for people to check the test results later on
    - The Dev Plat team should now maintain the list of test cases to be pushed at each image we have

 * Pre-Installed image:
   - Available as well for all flavors we care
   - Could be the target for LAVA when testing Ubuntu LEB images
   - A jenkins job would define what an image is, and how it should be created (from a hwpack and rootfs)
   - Needs development to support these kind of images
   - Pre-seed ubuntu images might also be an option
   - Would need to force to accept the license with the pre-hook script at lmc (accepting the license as debkeys)

 * State of the test cases
  - Lava-test is the way to go when integrating and creating new test cases
  - out of tree tests are possible (for private tests)
  - any important test for linaro should all go inside lava-test

 * Test result probe with rpc
  - View that returns a json with the test results
  - What kind of output: test results and link with log
  - Do basically what android is already doing

 * How to handle manual test reports
  - Prototype already available
  - Still useful and something we want to support

 * Easy way to submit test jobs per images
  - would be nice to have a way to request the latest version of an image/hwpack rather than having to specify
    https://bugs.launchpad.net/bugs/920350
  - web UI - View with image lists and test cases

 * Boot Loader
  - Hard to support, no real solution atm
  - Hardware solution (dual sd card, usb sd writer, jtag)

 * Master images
  - Update as needed
  - Maybe new features at u-boot could request it
  - Or critical bugs at the kernel level

(?)

Work Items