Update monthly toolchain benchmark process
The current process we use for monthly benchmarks needs some improvements.
= Background
We currently use Android's benchmark.git to help do some monthly benchmarking of different Linaro toolchains (4.5, 4.6, etc). We effectively build the benchmarks for each toolchain and then run them on a single Android image for Panda that month.
There are a few problems with this process:
1) benchmark.git is really more designed to benchmark a single toolchain, not multiple ones like we do.
2) benchmark.git includes headers/linker/libs built from an Android 4.4 toolchain. We should really be building the benchmarks using the versions of these files as produced by the toolchain we are benchmarking.
3) at runtime the benchmarks will use the libc/libm, etc provided the Android build for that month. So there's potential that a benchmark that might use a lot of math functions should really be using the libm that was built with that toolchain.
= Proposal
1) move benchmarks into actual android build
2) run the tests using lava-android-test
3) the output of two can be uploaded to the validation server
4) we can then produce reports using the data pushed to the validation server
Whiteboard
Meta:
Roadmap id: TCWG2011-GCC-O3
NOTES:
1) This will affect some of fgiff's work on: https:/
2) The android-built to validation server lava job seems to be pretty flaky. I will try doing the jobs this way. However, it may work better doing them locally this month. Also, since the builds really require HW acclerated graphics, you don't get great performance on the images we run in the validation lab.
Headline: Linaro's monthly toolchain benchmark process has been updated.
Acceptance: Linaro's monthly toolchain benchmark process has been updated.
Work Items
Work items:
Get skia_bench into android builds: DONE
Get v8 bench marks into android builds: DONE
Get 4.0.3 building with android prebuilt toolchain: DONE
Get 0xbench benchmark into android build: DONE
Dependency tree
* Blueprints in grey have been implemented.
