Include BUILD-INFO.txt in the Android builds

Registered by Zygmunt Krynicki on 2012-07-02

Why:
The infrastructure team has modified the build click-through to read a BUILD-INFO.txt file in the out directory of an Android build and to use its contents to drive the information in the click-through instead of hardcoding this text based on the build links.

Who:
Zygmunt Krynicki (the git repository and the licenses)
Gesha (test builds and hooking that up to jenkinks)

Where:
1) New overlay for open and open-restricted builds and updated overlays for origen and snowball
2) Jenkins configuration (TBD)
3) Additional scripts and source to all of my work is: https://github.com/zyga/android-build-info downloads for all the overlays: https://github.com/zyga/android-build-info/downloads

Context:
No card, general platform work.

Blueprint information

Status:
Started
Approver:
Zach Pfeffer
Priority:
High
Drafter:
Zygmunt Krynicki
Direction:
Approved
Assignee:
None
Definition:
Approved
Series goal:
Proposed for 2012q2
Implementation:
Started
Milestone target:
milestone icon backlog
Started by
Zygmunt Krynicki on 2012-07-03

Related branches

Sprints

Whiteboard

[zkrynicki, 2012-07-02]: Drafting based on https://docs.google.com/a/linaro.org/document/d/1e1D1AR8VhcfG-EnZhztWzBNJh7q_lSMfasH858BMScM/edit
[zkrynicki, 2012-07-09]: Priority redefined in context to my other blueprints for 12.07, starting work in exclusive mode
[zkrynicki, 2012-07-09]: Discussed design with bhoj, won't need another git repo, reflected this in blueprint summary
[zkrynicki, 2012-07-10]: Woot, I got it working, I'll patch other vendor taballs and ask maintainers to sync them. Android.mk that works: http://paste.ubuntu.com/1084994/
 (you'll need to have BUILD-INFO.txt.open in the same directory or have a valid /vendor/boot-info.mk. I'll put this in a git repository after I discuss with the team tomorrow. UPDATE: zach insisted on using overlays for everything so no there will be no git tree, just a tarball, that I'll publish, that defines /vendor/open/build-info/Android.mk. I'll ask snowball and origen maintainers to patch their tarballs with identical Android.mk and apprpriate license for their binary bits. Once that happens I'll edit build descriptions to see if we have everything done. I'll need to sync with pfalcon to know if jenkins won't choke on those new files though
[zkrynicki, 2012-07-11] Synced with Asac. Described current situation. Will add validation scripts on asac's request. Synced with Danilos, currently the backend that consumes BUILD-INFO.txt is being rewritten from PHP to Python+Django. There is some concern over snowball vendor tarball as it needs to have some files open and some files restricted and the backend may have incorrect checking order (indeterministic)
[zkrynicki, 2012-07-12] Some updates: the git repo with the open version is on github (https://github.com/zyga/android-build-info). I'll create branches for each vendor and export a tarball out of that.
[zkrynicki, 2012-07-12] I wrote 'verify-vendor-tarball' to check if the Android.mk makefile is correct. You can get the script here: https://raw.github.com/zyga/android-build-info/master/verify-vendor-tarball
[zkrynicki, 2012-07-13] Filed an RT ticket to create an open overlay: [rt.linaro.org #536] AutoReply: Please install new open overlay for android-build.linaro.org
[zkrynicki, 2012-07-13] While waiting for the ticket to settle I've started local test builds for all boards (no variations, just the main types we advertise) with my build-info additions. Results after weekend.
[zkrynicki, 2012-07-16] After some RT/IS woes we finally have the open tarball in the right spot
[zkrynicki, 2012-07-16] There are some build issues that I'm trying to understand that prevent the build from starting (Infrastructure, unrelated to my changes). Debugging now: https://android-build.linaro.org/jenkins/job/linaro-android_panda-ics-gcc47-tilt-tracking-blob/125/parsed_console/?.
[zkrynicki, 2012-07-16] Blocked by https://bugs.launchpad.net/linaro-android/+bug/1025311
[zkrynicki, 2012-07-17] Unblocked by that and another bug (only snowball and origen overlays were copied to the build machines, now danilos added 'open' as well).
[zkrynicki, 2012-07-18] Panda builds with the open overlay are working now. It's still unsure if the BUILD-INFO file is actually copied to the apache download site: https://android-build.linaro.org/builds/~linaro-android/panda-ics-gcc47-tilt-tracking-blob/#build=131
[zkrynicki, 2012-07-18] Gesha is back from leave, we'll try to sort out the remainder of the problems and update all vendor overlays this week.
[zkrynicki, 2012-07-18] Reported a bug on igloocommunity to include build-info in the vendor tarball. Unable to assign that to Mathieu Poirier (lauchpad said 'constraint bla bla bla'). https://bugs.launchpad.net/igloocommunity/+bug/1026101
[zkrynicki, 2012-07-18] It seems that deployment may be blocked by staging servers for snapshots. The RT tickets for that are 517 and 504. The latter ticket is 'delayed' till 23rf of July. This puts this blueprint at risk
[zkrynicki, 2012-07-18] Samsung vendor tarball is maintained by Annamalai
[zkrynicki, 2012-07-18] Got two new tarballs from mpoirier (mali and mm), testing on hackbox now. If successful I'll file an RT ticket to update our current snowball blobs and do a test build.
[dzin July 18, 2012] Please add a Headline and Acceptance fields.
[zkrynicki, 2012-07-19] Both (mm, mali) builds for snowball work OK. Requesting RT ticket
[zkrynicki, 2012-07-19] Filed RT ticket [rt.linaro.org #538] AutoReply: Please add two new overlays for snowball (20120718)
[zkrynicki, 2012-07-19] Ticket 538 done but reopened again (the files have incorrect permissions and cannot be downloaded, there is no EULA)
[zkrynicki, 2012-07-19] Requested ticket [rt.linaro.org #540] AutoReply: Add test overlays to snapshots.linaro.org. This ticket will allow us to test builds with fake overlays (correct license but no binary blobs) before going live with real builds.
[zkrynicki, 2012-07-20] Snowball and Panda builds confirmed working (build finished properly, BUILD-INFO.txt copied properly, device boots and works). No origen tests so far.
[zkrynicki, 2012-07-23] Blocked by RT ticket 540
[pfefferz, 2012-07-23] Moving to 12.08 because its blocked on RT ticket 540
[zkrynicki, 2012-07-23] Registered followup blueprint as: https://blueprints.launchpad.net/linaro-android/+spec/deploy-build-info-txt
[asac, 2012-07-26]: this blueprint is not implemented. it doesn't deliver the aimed for headline/acceptance. we don't have build info everywhere and the new click through approach is not used for everything binary.
[asac, 2012-07-26]: has to go through post-mortem. proposed split up: make this blueprint solely about rolling test builds and bringing up first iteration of overlay examples; make roll out to all tip builds and rolling out the final infrastructure tweaks separeate blueprints for 11.08.
[pfefferz, 2012-07-26] That's already been done. The split was complete on Monday at your request. Take a look at 12.08 specifically, https://blueprints.launchpad.net/linaro-android/+spec/deploy-build-info-txt.
[zkrynicki, 2012-07-27] Moving to backlog, acked by pfefferz

Headline: Binary Blobs use new Data Driven Click-through
Acceptance: All builds use BUILD-INFO.txt

Download links for the security overlays (without the binary parts obviously)
https://github.com/zyga/android-build-info/downloads

(?)

Work Items

Work items:
Get the BUILD-INFO.txt’s from Georgy for each baseline: DONE
Talk to Gesha about his test build that should get the BUILD-INFO.txt in the right location in the build: DONE
Create a generic Android.mk for the license stuff: DONE
Create an overlay for open: DONE
Create a script that checks the license of a vendor tarball: DONE
Create a script that validates Android.mk embeded in a vendor tarball : DONE
Work around the django/PHP rewrite and indeterministic matching problem: DONE
Create an overlay for open restricted: DONE
Create an updated overlay for snowball: DONE
Create an updated overlay for origen: DONE
Install open overlay on android-build.linaro.org: DONE
Debug failure of build 125 (panda ics open): DONE
Patch build system to allow one to fetch overlays from the web: POSTPONED

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.