Deploy BUILD-INFO.txt in specified official builds (all builds)

Registered by Zygmunt Krynicki on 2012-07-23

Why:
In the 12.07 cycle we have implemented support for keeping BUILD-INFO.txt (a description of access protection and optional EULA associated with binary downloads) along with the vendor binary blobs. This simplifies the infrastructure and removes the need to maintain fragile naming patterns. While coding is complete we still need to deploy this to all of our production builds to reap the benefits.

Who:
Zygmunt Krynicki

Where:
1) Jenkins configuration for each build
2) snapshots.linaro.org, specifically http://snapshots.linaro.org/android/binaries/

Context:
No card, general platform work.

Blueprint information

Status:
Started
Approver:
Zach Pfeffer
Priority:
High
Drafter:
None
Direction:
Needs approval
Assignee:
None
Definition:
Drafting
Series goal:
Proposed for 2012q2
Implementation:
Blocked
Milestone target:
milestone icon backlog
Started by
Zygmunt Krynicki on 2012-07-23

Related branches

Sprints

Whiteboard

[zkrynicki, 2012-07-23] Moving remaining work items from https://blueprints.launchpad.net/linaro-android/+spec/include-build-info-txt
[zkrynicki, 2012-07-23] Updated origen overlay now deployed on snapshots.linaro.org (ticket 538), IS has corrected EULA settings as the new binary blobs use different pathname patterns (20120718/{mm,mali}/vendor.tar.bz2)
[zkrynicki, 2012-07-23] Started fakeblob build for origen: https://android-build.linaro.org/builds/~zkrynicki/origen-ics-gcc47-samsunglt-stable-fakeblob-build-info/
[zkrynicki, 2012-07-23] "restricted" overlay is now technically installed (as tests/*/restricted.tar.gz). It's "probably" enough as the only use case that I know is the restricted big.LITTLE switcher build with non-public git trees. Since there are no binary blobs there there is no existing overlay so this restricted one should work just as well. I need to consult with asac / infra before testing such builds though.
[zkrynicki, 2012-07-24] Using panda build with source overlays for all tests. This avoids build errors that otherwise arise (some parts of the overlay are essential for the build).
[zkrynicki, 2012-07-24] Initial tests seem to indicate that BUILD-INFO.txt is still ignored and name pattern (name of the build) still overrides all other settings. I've started another build under 'neutral' name to confirm: https://android-build.linaro.org/builds/~zkrynicki/bummer-no-license-to-guess/
[zkrynicki, 2012-07-25] I've sent an email about the failed builds to the mailing list and all the stakeholders / essential engineers ( http://lists.linaro.org/pipermail/linaro-android/2012-July/001359.html ). In the morning I'll see what we can do to fix the problems.
[zkrynicki, 2012-07-25] I've synced with gesha about the current implementation and our BUILD-INFO.txt files. We've also spotted two issues with the previous tests: 1) the overlay was specified twice which confused me in reading the results back 2) the snowball and origen overlay used *.tar.gz file pattern. This is now fixed in git, the new pattern is *.tar.* which should hopefully be future-proof. I'll request a new upload of test overlays. This also means that real snowball overlay we've uploaded earlier is incorrect and any build done with it will not show the EULA correctly.
[zkrynicki, 2012-07-25] Requested: [rt.linaro.org #550] AutoReply: Upload two overlay tarballs to snapshots.linaro.org (2012-07-25)
[zkrynicki, 2012-07-26] Testing and gesha confirms that BUILD-INFO.txt is _ignored_ unless the build configuration includes the word 'blob' in it. This cannot be fixed until the rollout of staging servers and successful upgrade from old php codebase to new django codebase.
[zkrynicki, 2012-07-26] Running tests on configuration with 'blob' in the name https://android-build.linaro.org/builds/~zkrynicki/build-info-test-with-the-b-word-in-name-blob to see if we can migrate snowball and origen.
[zkrynicki, 2012-07-26] Builds fail with OOM killer killing things. Gesha is investigating. Similar situation is currently affecting ICS panda and JB nexus (but not panda). Unclear as for why this is occurring now
[zkrynicki, 2012-07-26] Blocked by https://bugs.launchpad.net/linaro-android-infrastructure/+bug/1028561
[zkrynicki, 2012-07-27] Tests show that 'blob' is not sufficient. Gesha confirmed this and checked the code. We need both 'blob' and either 'origen' or 'snowball'. Aborting further tests with PHP codebase. Testing with Django codebase (revno 86)
[zkrynicki, 2012-07-27] Tests with django codebase were positive. It seems that all protection systems work correctly. I could not finish openid testing as my localhost application was not whitelisted to reveal appropriate group membership (still the download to affected files was rejected). Testing also found that the *.tar.* pattern does _not_ protect *.img files which could be a problem for uboot and later full-system images as prepared for fastboot.
[zkrynicki, 2012-07-27] We had a sync meeting (asac, pfeffers, gesha, jamestunnicliffe and me) to address the roadmap of this blueprint. We've decided to postpone all further work until Django rewrite lands in production. I will still file RT tickets to remove the broken snowball uploads and to send my locally repackaged vendor tarballs back.
[zkrynicki, 2012-07-27] Moving this task to backlog
[zkrynicki, 2012-07-27] Last update: I've removed the incorrect snowball vendor tarballs, requested new tarballs to be uploaded (for both snowball and origen) via RT ticket 555. I've patched the branches at https://github.com/zyga/android-build-info/tree/snowball and https://github.com/zyga/android-build-info/tree/origen and have removed the old (v2) tarballs from https://github.com/zyga/android-build-info/downloads. The only difference is this patch: https://github.com/zyga/android-build-info/commit/912c69e2a64b7a86709cc864fe6deb3405f785bb which as been applied to both origen and snowball branches.

Quick status:
Sending to backlog until django deployment lands

Headline: TBD
Acceptance: TBD

(?)

Work Items

Work items:
Install test overlays on snapshots.linaro.org (RT ticket 540): DONE
Run additional tests under different names as previous tests suggest name pattern still overrides BUILD-INFO.txt: DONE
Run four private test builds with test overlays to validate license settings (open): DONE
Upload fixed test/mock overlays for snowball and origen (RT ticket 550): DONE
Run four private test builds with test overlays to validate license settings (snowball): DONE
Run four private test builds with test overlays to validate license settings (restricted): DONE
Run four private test builds with test overlays to validate license settings (origen): DONE
Figure out which part of the build/download system is incorrect (with Gesha): DONE
Test BUILD-INFO.txt on new Django codebase: DONE
Remove incorrect snowball vendor tarballs (RT ticker 554): DONE
Re-install updated/fixed overlay for snowball on snapshots.linaro.org: DONE
Install updated overlay for origen on snapshots.linaro.org: DONE
Install restricted overlay on snapshots.linaro.org: DONE
Enumerate all the build configurations that need patching: POSTPONED
Patch each configuration to use new overlay: POSTPONED
Test each build: POSTPONED

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.