Tools for Extracting the Results from ARM Benchmark APKs have Been Written
Why?
Benchmark APKs requested by ARM currently output their results to the screen. Tools need to be written to extract the benchmarks so they can be processed.
Context?
This Blueprint is in support of the benchmarking goals that ARM set forth during the 12.06 Connect, it should allow us to work with the results easier and collate the results with other results on a per build basis.
What gets produced?
A set of Monkeyrunner scripts (or other extraction method), perhaps a script per benchmark, that can read the output screen of the APKs we don't have access to and extract the results into a common form, like a JSON text file.
Where will the work get put?
The host scripts should get put in a git like, MONKEY_
Blueprint information
- Status:
- Complete
- Approver:
- Zach Pfeffer
- Priority:
- High
- Drafter:
- vishal
- Direction:
- Approved
- Assignee:
- Yongqin Liu
- Definition:
- Approved
- Series goal:
- Accepted for 2012q3
- Implementation:
-
Implemented
- Milestone target:
-
12.11
- Started by
- Bernhard Rosenkraenzer
- Completed by
- vishal
Related branches
Related bugs
Sprints
Whiteboard
Notes:
[2012/7/2 pfefferz] Please put notes here.
[2012/7/23 pfefferz] Since JB landed mid-cycle all of Bero's work was put on hold
[2012/8/21 pfefferz] Putting back on the schedule for 12.09.
[2012/8/28 vishalbhoj] We have added a hook to dump the result from textview by setting "ro.debug.textview" to true . We are able to dump results for andebench,
[2012/09/05 vishalbhoj] There is a mistake in the above note. We can capture results for nbench with the textview hook . This doesn't work for geekbench.
[2012/09/05 berolinux] Chances are we just need another hook for capturing what they're displaying. I'd start with android.
[2012/09/17 liuyq0307] result of nbench on official build: https:/
[2012/09/20 liuyq0307] the glbenchmark test needs the connection to internet via wifi. but this is not supported in lava now. so we need to wait to check the result from android-build
[2012/09/24 pfefferz] Bumping to 12.10 due to blocking issues (WiFi in LAVA lab).
[2012/10/03 liuyq0307] add the dependency bp information https:/
[2012/10/07 liuyq0307] result of glbenchmark/
[2012/10/09 pfefferz] Vellamo results at https:/
[2012/10/15 liuyq0307] Vellamo results at customized build https:/
[2012/10/16 liuyq0307] The latest lava-dispatcher and lava-android-test source will be deployed to the lava product instance by this weekend.
[2012/10/22 pfefferz] Bumping to 12.11 while the remaining issues are worked out.
[2012/10/25 liuyq0307] Vellamo result on official build: https:/
[2012/10/25 liuyq0307] Test result for origen stable: https:/
[2012/10/25 liuyq0307] Test result for snowball stable: https:/
Meta:
Headline: Tools for extracting the results from ARM benchmark APKs have been developed.
Acceptance:
1. A mechanism to extract the result of each benchmark has been written
2. The mechanism have been integrated into LAVA
3. The output of each run is matched to format that can be further processed like a LAVA bundle or text file.
Roadmap id: CARD-134
Work Items
Work items:
[vishalbhoj] Check if the benchmark APKs have any possibility to write their output anywhere except on screen: DONE
[vishalbhoj] If so, enable it: DONE
[vishalbhoj] If not, come up with a mechanism to extract the results in a useful format anyway: DONE
[vishalbhoj] Add a hook to TextView to dump results: DONE
[vishalbhoj] Implement it: DONE
[vishalbhoj] Make the results available to LAVA: DONE
[liuyq0307] Implement for GLBenchmark: DONE
[liuyq0307] Implement for nbench: DONE
[liuyq0307] Implement for vellamo: DONE
[liuyq0307] Implement for geekbench: DONE
[liuyq0307] Confirm for GLBenchmark from android-build: DONE
[liuyq0307] Confirm for vellamo from android-build: DONE
[liuyq0307] Confirm for geekbench from android-build: DONE
[liuyq0307] Confirm for quadrant from android-build: DONE
[liuyq0307] Extend Canvas.drawText or similar to get text for quadrant: DONE
Make sure WiFi is working in the lab: DONE
[liuyq0307] Modify lava-android-test to deal the fail pattern for vellamo: DONE
Dependency tree
* Blueprints in grey have been implemented.
