Linaro Android

Add Monthly Toolchain Benchmarking to Lava

Registered by Andy Doan

This blueprint has been superseded. See the newer blueprint "Update monthly toolchain benchmark process" for updated plans.

The monthly toolchain benchmarks:
  https://wiki.linaro.org/Platform/Android/AndroidToolchainBenchmarking

are almost completely automated. We need to figure out the missing pieces and get this running in the validation farm.

A discussion took place at Connect on this topic. The notes are here:
  https://blueprints.launchpad.net/linaro-android/+spec/linaro-platforms-lc4.11-android-benchmarking

Blueprint information

Status:
Complete
Approver:
Zach Pfeffer
Priority:
High
Drafter:
Andy Doan
Direction:
Approved
Assignee:
Andy Doan
Definition:
Superseded
Series goal:
Accepted for future
Implementation:
Implemented
Milestone target:
milestone icon 12.01
Started by
Frans Gifford
Completed by
Andy Doan

Related branches

Sprints

Whiteboard

Meta:
Roadmap id: TCWG2011-GCC-O3

Notes:
A rough outline of the work:
*Get LAVA to run the benchmark*
Add a test to lava-android-test that can benchmark a single toolchain. Pass the URL of the toolchain as a parameter to this test. This test will need to do something similar to the run-benchmarks script in doanac's linaro-toolchain-benchmarking git tree.

*Get android-build.l.o to trigger the benchmark*
Modify the build-android-toolchain build script to make it possible to submit the toolchain to LAVA for benchmarking. This should be similar to how the build-android script submits builds to LAVA for testing.

[fgiff, 20111116] This isn't going to complete this cycle, so it will need to be split.

[fgiff, 20111123] Blocked on https://blueprints.launchpad.net/lava-android-test/+spec/support-parameterized-tests

[fgiff, 20111125] Caught up with mwhudson on irc yesterday (http://irclogs.linaro.org/2011/11/24/%23linaro.html) and got the details of how to write parameterised tests.

[dzin, 2011/11/25] Moving to 11.12

[fgiff, 20111130] Assigned some bits to Amit. The changes in a-b.l.o are dependent on getting the benchmark into LAVA first.

[fgiff, 20111206]
Moved the reporting out into a separate BP.

[mansson 2011-12-21] This one: https://blueprints.launchpad.net/linaro-android/+spec/linaro-android-bmark-process
[dzin 2011-12-22] Retarget to 12.01

[asac 2012-01-02] lets talk about implementation with plars; we have to see how and if we can keep the data for this benchmark public; I see a chance to use a special/fixed "default" board type that is then used in the result data to work around.

[mansson 2012-01-26] This BP is superseded, and the new (very similar) BP is DONE. I am therefore marking this BP as DONE too.

Headline:
The monthly Linaro Android toolchain benchmark results are automatically generated by LAVA.

Acceptance:
The "monthly benchmark" that has been output to https://wiki.linaro.org/Platform/Android/AndroidToolchainBenchmarking is done per build as a LAVA job.

(?)

Work Items

Work items:
Run through test procedure locally: DONE
Work out how to run the procedure on LAVA: DONE
Write code to run the procedure in LAVA: DONE
Get benchmark integrated into LAVA: DONE
Get benchmark deployed at validation.l.o: DONE

Dependency tree

* Blueprints in grey have been implemented.

Nearby

This blueprint contains Public information 
Everyone can see this information.