Continuous Integration Kernel Testing in LAVA

Supporting large scale testing of kernel trees and defconfig in the LAVA lab is a high priority engineering effort across all Linaro engineering and Landing Teams for Q4.10. This organization wide effort imposes a growing list of requirements that LAVA team has started to work on during Q3 and will focus on driving forward during the Q4.2011.

This effort involves
improvements to the lava dashboard to ensure that the build and test results data can be efficiently consumed by kernel engineers
improvements to the lava dashboard to make it easy to follow every step of a submitted job from the scheduler to the job submissions step.
increase of capacity for all LEB board types deployed in the lava-lab to a level suitable to continuously track 20+ kernel branches with multiple defconfigs each.
improvements to lava job submissions tools that make it easy for services like the kernel CI service to hook link from their build page to the scheduler job and the results.

The goal of this session is to discuss user stories around LAVA components for driving continuous integration testing and results tracking through LAVA.

Session Notes:
Currently the kernel is cross built in ec2 by jenkins -- build results are submitted to the dashboard and if successful a test is run in LAVA.
http://validation.linaro.org/lava-server/kernel-ci-views/index attempts to convey a summary of the results.
Current process of getting your own tree is manual - talk to Deepti to get your tree included.
Wishlist:

    Want to able to use the set up topic trees for temporary testing.

    Maybe need to make it easier for KWG devs to set up trees for themselves

    We might want to use per-user configs so that users can select the trees/configs they care about

    Per-user reports and a web form so they can submit git tree, branch and config for a one-time or regular testing of their tree

    per-tree specification of tests to run

    Determine which tests to run with the new kernel: ACTION: mwhudson to submit mp to change this

    ltp

    hw enablement tests

    PM functional tests

    Store the kernel version (uname) so that we can cross-reference this

    Build results notification (by email, RSS feed)

    Can just do this in Jenkins for now

    Want to do this in LAVA

    RSS ok for now

    Weekly build-status digest email for the community? Deepak to do this manually for now

    John's builds don't appear on the tree view

    Waterfall view perhaps not appropriate for one-time builds

    Should use tags for grouping results (now that tags exists)

    deepti to change jenkins jobs to use tags

    Add tagging support to dispatcher so the results get tagged similarly

    View for test/build combination

    Easy way to see last successful build for a test/build

    Waterfall-ish view of test suite vs test/build history

    Also improve bundle view

Anti-wishlist:

    Builds of out of tree configs

    Basically, developer should commit the Kconfig changes to the tree

ACTION: start tagging build and test results (with LAVA tags) Figure out how our read-only tags look like (since tags cannot be changed afterwards).
Mouse-over git-describe: show the shortlog