CI with the Linux Linaro packages

[asac, 28 Sep, 2011]: needs progress by end of day, otherwise this is not getting done by 11.09 and goes through post-mortem.
[rsalveti, 29 Sep, 2011]: Not fully integrated, but the build scripts were all done already, now we just need to integrate it with jenkins and lava.
[dzin, 3 Oct, 2011]: For 11.09 the build scripts have been done, but the integration with jenkins and LAVA will be broken out into a new blueprint targeted for 11.10.
[dzin 20 Sep, 2012] remove milestone info.

Headline: The Linux Linaro packages are now generated and validated at a continuous integration loop.
Acceptance: Build bot in place, build at every new commit or at least daily. Results from the merge published at a staging git tree. Packages generated, with hwpack in place and tested at Lava.

Draft and discussions:
 - Work implemented at a build bot (jenkins? need to check)
 - Separate sauce + packaging + config into a different tree
   - Packaing is easily separated because it all lives in debian and debian.linaro
   - Sauce is a different issue since it touches files all over the kernel tree so:
     - What does separate sauce mean?
       - A separate head that has sauce applied? This could just be a clone (or modified clone) of a Ubuntu tree.
       - A directory containing sauce extracted from Ubuntu with git format patch?
    - Extracting Ubuntu sauce can be done by a human periodically and put in a separate head. Then the CI script will just merge or replay (via rebase with appropriate arguments)
 - Scripts that merge both trees and produce a tree to be built
 - Should this tree be pushed somewhere?
    - The staging tree should have the state to remember what inputs went into it (packaging commit id and upstream commit id).
    - This state could be in tag, in topmost commit message or in debian.linaro/changelog.
 - Check:
   - ABI
     - 3 steps: no check, manual check and bump, automated bump
   - Config changes
 - Sauce:
   - Steps:
     - No sauce for the beginning
     - Entire sauce
     - Drop the sauce we don't need (x86 only)
 - Deepti is helping with the Jenkins part
 - Cross build for every Kernel in CI
 - Per <period> an upload of the resulted branch
   - Source package
   - Build natively
 - How to report failures?
   - Config conflicts
   - ABI bump
   - Rebase conflict
   - Build failure
   - Deploy failure?
 - HWpack is produced
 - Tested at LAVA

Work items:
Sync with Deepti and check what is needed from our side to make the script available at Jenkins, and understand the Jenkins requirements: TODO
[rsalveti] Sync with Deepti to understand how the hwpack is created from the build, and what is needed to fire up new tests at Lava to validate the kernel: DONE
Create script to automatically rebase and build existing packaged kernel onto updated upstream kernel: INPROGRESS
Create per flavor packaging/config trees from the existing omap-only, mx51-only... trees/tags without sauce: TODO
Create script to merge current head or specific commit or tag of linux-linaro with a particular flavour packaging and build: TODO
Create a git tree at git.linaro.org that should be updated with the merge results: TODO
Change script to tag and push the resulted changes to the staging tree (independent of success or failure of build): TODO
Add config seed files for configs for satisfying ubuntu enforce: DONE
Add config seed files for configs for standard Linaro features: DONE
Add config seed files for configs for per flavour/soc: DONE
Create kernel build scripts to create configs from seed files: DONE
Make the build script to compare the config result to previous build and report changes but do not fail the build: TODO
Add ability to apply sauce (from directory of patches or git tree and commit id's of beginning and ending of sauce): TODO
Improve the build script to detect abi change, bump abi and restart build: TODO
Create script for assisting in creation of sauce tree or dir: TODO
Add filtering to sauce creation script: TODO
Each friday, create source pkg from latest successful build and upload to ppa: TODO