Tools for Extracting the Results from ARM Benchmark APKs have Been Written

Registered by Bernhard Rosenkraenzer on 2012-07-02

Why?
Benchmark APKs requested by ARM currently output their results to the screen. Tools need to be written to extract the benchmarks so they can be processed.

Context?
This Blueprint is in support of the benchmarking goals that ARM set forth during the 12.06 Connect, it should allow us to work with the results easier and collate the results with other results on a per build basis.

What gets produced?
A set of Monkeyrunner scripts (or other extraction method), perhaps a script per benchmark, that can read the output screen of the APKs we don't have access to and extract the results into a common form, like a JSON text file.

Where will the work get put?
The host scripts should get put in a git like, MONKEY_RUNNER_URL_1="git://android.git.linaro.org/test/linaro/android/system.git"

Blueprint information

Status:
Complete
Approver:
Zach Pfeffer
Priority:
High
Drafter:
vishal
Direction:
Approved
Assignee:
Yongqin Liu
Definition:
Approved
Series goal:
Accepted for 2012q3
Implementation:
Implemented
Milestone target:
milestone icon 12.11
Started by
Bernhard Rosenkraenzer on 2012-07-04
Completed by
vishal on 2012-10-25

Related branches

Sprints

Whiteboard

Notes:
[2012/7/2 pfefferz] Please put notes here.
[2012/7/23 pfefferz] Since JB landed mid-cycle all of Bero's work was put on hold
[2012/8/21 pfefferz] Putting back on the schedule for 12.09.
[2012/8/28 vishalbhoj] We have added a hook to dump the result from textview by setting "ro.debug.textview" to true . We are able to dump results for andebench,Antutu,Linpack,CaffeineMark,GLBenchmark for now. This approach doesn't work for Vellamo,nbench and quadrant since they don't use textview to display the results.
[2012/09/05 vishalbhoj] There is a mistake in the above note. We can capture results for nbench with the textview hook . This doesn't work for geekbench.
[2012/09/05 berolinux] Chances are we just need another hook for capturing what they're displaying. I'd start with android.graphics.Canvas.drawText
[2012/09/17 liuyq0307] result of nbench on official build: https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-stable-blob/#build=65
[2012/09/20 liuyq0307] the glbenchmark test needs the connection to internet via wifi. but this is not supported in lava now. so we need to wait to check the result from android-build
[2012/09/24 pfefferz] Bumping to 12.10 due to blocking issues (WiFi in LAVA lab).
[2012/10/03 liuyq0307] add the dependency bp information https://blueprints.launchpad.net/lava-lab/+spec/connect-wifi-aps-to-exteranal-net
[2012/10/07 liuyq0307] result of glbenchmark/geekbench/quadrant https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-stable-blob/#build=90
[2012/10/09 pfefferz] Vellamo results at https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-stable-blob/#build=90
[2012/10/15 liuyq0307] Vellamo results at customized build https://android-build.linaro.org/builds/~liuyq0307/jb-panda-stable/#build=17
[2012/10/16 liuyq0307] The latest lava-dispatcher and lava-android-test source will be deployed to the lava product instance by this weekend.
[2012/10/22 pfefferz] Bumping to 12.11 while the remaining issues are worked out.
[2012/10/25 liuyq0307] Vellamo result on official build: https://android-build.linaro.org/builds/~linaro-android/panda-jb-gcc47-tilt-stable-blob/#build=102
[2012/10/25 liuyq0307] Test result for origen stable: https://android-build.linaro.org/builds/~linaro-android/origen-jb-gcc47-samsunglt-stable-blob/#build=126
[2012/10/25 liuyq0307] Test result for snowball stable: https://android-build.linaro.org/builds/~linaro-android/snowball-jb-gcc47-igloo-stable-blob/#build=117

Meta:
Headline: Tools for extracting the results from ARM benchmark APKs have been developed.
Acceptance:
1. A mechanism to extract the result of each benchmark has been written
2. The mechanism have been integrated into LAVA
3. The output of each run is matched to format that can be further processed like a LAVA bundle or text file.
Roadmap id: CARD-134

(?)

Work Items

Work items:
[vishalbhoj] Check if the benchmark APKs have any possibility to write their output anywhere except on screen: DONE
[vishalbhoj] If so, enable it: DONE
[vishalbhoj] If not, come up with a mechanism to extract the results in a useful format anyway: DONE
[vishalbhoj] Add a hook to TextView to dump results: DONE
[vishalbhoj] Implement it: DONE
[vishalbhoj] Make the results available to LAVA: DONE
[liuyq0307] Implement for GLBenchmark: DONE
[liuyq0307] Implement for nbench: DONE
[liuyq0307] Implement for vellamo: DONE
[liuyq0307] Implement for geekbench: DONE
[liuyq0307] Confirm for GLBenchmark from android-build: DONE
[liuyq0307] Confirm for vellamo from android-build: DONE
[liuyq0307] Confirm for geekbench from android-build: DONE
[liuyq0307] Confirm for quadrant from android-build: DONE
[liuyq0307] Extend Canvas.drawText or similar to get text for quadrant: DONE
Make sure WiFi is working in the lab: DONE
[liuyq0307] Modify lava-android-test to deal the fail pattern for vellamo: DONE

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.