Better Android Build Service
Improve http://
Blueprint information
- Status:
- Complete
- Approver:
- James Westby
- Priority:
- Medium
- Drafter:
- Paul Sokolovsky
- Direction:
- Approved
- Assignee:
- Paul Sokolovsky
- Definition:
- Approved
- Series goal:
- Accepted for 2011q2
- Implementation:
-
Informational
- Milestone target:
- None
- Started by
- James Westby
- Completed by
- James Westby
Whiteboard
User stories
1. As a build system maintainer I want to be able to quickly deploy latest build system changes in a sandbox environment So the changes can be tested and debugged before deploying to production.
Should elaborate existing setup and deployment scripts so it was easy to install complete cloud-buildd system on a separate machine (instance), as well as update existing deployment. This is already work in progress and on critical path to most of other user stories and improvements.
https:/
2. As a EC2 Administrator I want that cloud-buildd consumed instances and their machine time optimally So that we avoided system overload/
Need to put up a standalone cronjob which will monitor, notify about, and terminate about too many or too long-running build slave instances. Also, need to make build instances to shutdown as soon as build is finished. Current idea is to use post-build plugin to cause instance self-shutdown. But need to be prepared to patch Jenkins EC2 plugin too.
https:/
3. As an Android engineer I want to quickly build Android images so that I can continue with further development, fix any issues ASAP, proceed with testing, or hand clean-room build to another party.
Per the current performance stats, 75% of build time is spent in the Android make per se, so if we want to improve build time, we should target compilation time. It will be hard to cut down compilation time on Android make system level, so we need plug into other stages, like uses systems like ccache/dist-cc. Of these, ccache appears to be better candidate for us.
https:/
4. As an Android engineer I want to build other types of Android artifacts besides target images (e.g., toolchain) So that I can use same process for various Android-related tasks
As an Engineer I want to build different (not Android related) type of artifact So that I can benefit from cloud-buildd features.
As an Android engineer I want to add override manifest (local_
Work is ongoing.
https:/
5. As an Android engineer I want to perform clean-room, 100% reproducible build of Android images so that I knew that my (or somebody else's) changes actually don't cause any issues and can be reliably used by other parties.
Need to find/debug/fix places where old data may be used, like recent case of mirror service failing, but not reporting error, so build went on to fail with obscure error later. Otherwise, need to implement original requirement: single build on single instance, no instance reuse (thus, this depends on ability to shutdown instance after the build).
6. As a Validation engineer I want to get notification of successful builds and query build metadata so I can start validating image and be able to record all needed data and crosslink them with other continuous integration subsystems
This depends on requirements passed from Validation team. It is expected that this will be an XMLRPC call (or script to execute) to be made at the end of a build.
7. As an Android engineer I want to quickly see high-level build report and build errors So that my productivity is improved.
As a casual linaro-cloud-buildd system user I want to have streamlined, clean, visible UI So that I can use it at all, more, and more productively.
Need to keep improving UI.
8. As an Android engineer I want to leverage build service cloud infrastructure, so I can develop in the cloud.
The idea here is that Android build takes time to set up and build itself, and build service provides nice few-clicks solution to get your build done in a reasonable time. We want to reuse this scheme for development work too, both in-house and as a service to provide to partners. Proposed interface is as following. Like local Android build starts with a command
repo init -u git://android.
then following command will start a dedicated instance, check out the source, and optionally start a build:
linaro-cloud-repo init -u git://android.
Issues: this apparently should be an EBS instance, so it could be stopped and started again. And anyway need ways to stop/garbage collect unused instances.
9. As an Android developer/bug fixer I want to reproduce the same build locally that was produced remotely, either on tip or in the past and I'd like to validate that is is indeed the same.
Produce exact revision pegged manifest as output of build. Fingerprint and manifest build artifacts.
10. As an image-consuming party I want to have access to build archive of sufficient depth So that I can perform regression and trend analysis, etc.
Publish build artifacts in S3, support 3rd party publishing credentials. Consider using enterprise build artifact repository system (make review of such systems).
----
Session notes (from http://
Welcome to Ubuntu Developer Summit!
#uds-o #track #topic
https:/
1. Highlight build failures
2. Reduce build time
3. New developer experience
4. Develop in the cloud
5. Lower local build time
6. Rewrite manifests to list the head commit of all sub-gits when the build completes
7. Local manifest upload and build
A: Support override manifests
Expected improvement areas:
* Stability
* Correctness
Build artifacts - fingerprinting and manifesting them
HEAD id manifest of source git trees
* Scope (extending)
Current: Android
Current: Toolchain
Want: Build Ubuntu
Want: Build Sub
Want: Benchmark
30 mins tip-to-tail
* Ease of use
Make build easy - smart scripts, easy instructions
* Performance
Bring down compile time: to fit into amazon accounting 1hr, etc. Improve android build scripts? Use ccache/distcc?
Reproduce cloud build locally (exact source revisions)
As a Validation engineer or other image-consuming party I want to get results of clean-room, 100% reproducible build of Android images so that I knew I'm working with reliable build results, easily trackable, and for which metadata like build configuration, date, logs, links to other builds is readily available.
As an image-consuming party I want to have well-known location for the latest (latest successful) build and its metadata So that I can script/automate my process involving the images
As an Android engineer or validation engineer I want to tag/annotate particular build with specific/arbitrary data So that I can specify that given build has particular characteristics and query them later
As a Android community member I want to see build results for various well-known Android trees besides Linaro's So that I had better perspective of what projects are running in the community, their status, and know that Linaro is good destination for all things Android.
As an Android developer/bug fixer I want to reproduce the same build locally that was produced remotely, either on tip or in the past and I'd like to validate that is is indeed the same. (Zach)
As an Android developer I want to make build instance to stay for a while after build completion So I can hack in working tree and resubmit it for validation (Zach)
As a Validation engineer I want to receive (programmatical) notification of build completion and status So I can trigger validation on it
As a Validation engineer I want to start a particular build programmatically So I can integrate build service into general continuous integration infrastructure
<asac> pfalcon: here another story: user tries a few builds. in order to know what he actually tried when looking at build history he wants to be able to pass a "comment/reason text" when hitting "build now" button ;)
<pfalcon> asac: ok, noted. do you want that comment to be shown in our frontend (not jenkins)?
<asac> pfalcon: frontend
Asac: group build (start few builds in parallel with single click)
Linaro Cloud Build Daemon
Paul presenting
Linaro-cloud-buildd
We use Amazon EC2 currently
We need a lot of compute for a little bit of time
Everything has been built for Ubuntu
Build service is parameterized
Jenkins EC2 Plugin
How can we improve?
Work Items
Dependency tree
* Blueprints in grey have been implemented.
