Investigate Android Benchmark Variance
Why?
The Toolchain Working Group has requested that we create a system where the run-to-run variance of our benchmarks is 1% for native and JAVA workloads. These low -variances will allow comparisons across baselines and allow benchmark guided optimization. An effort t olower this variance was done and the feedback from the TCWG was to look at the higher variances and see what's causing them.
Context?
This is part of the larger ARM benchmarking effort.
What gets produced?
A mechanism that runs before a benchmark that puts the system into a lower variance state han it was or a list of benchmarks who's variances cannot be characterized or lowered.
Where will the work get put?
A script on android.
Blueprint information
- Status:
- Complete
- Approver:
- Zach Pfeffer
- Priority:
- Medium
- Drafter:
- vishal
- Direction:
- Approved
- Assignee:
- vishal
- Definition:
- New
- Series goal:
- Accepted for 2012q3
- Implementation:
-
Implemented
- Milestone target:
-
12.09
- Started by
- Zach Pfeffer
- Completed by
- Zach Pfeffer
Related branches
Related bugs
Sprints
Whiteboard
Notes:
[2012/8/22 pfefferz] Please put notes here.
[2012/9/04 vishalbhoj] inputs from TCWG wrt signoff:
Benchmarks are consistent,with lower variance and we have explanations for higher variance .
Some of those with high variance were due to
* GC
* Database calls causing I/O
* Stream copy and stdlib copy
The A9 has higher variance on memory and branch predictor than the A8, should try on a BeagleBoard.
try memory benchmarks on cortex-A8 board (beagleboard).
Meta:
Headline: Native and JAVA benchmarks can be run with 1% variance and we can use benchmarks to optimize
Acceptance: 1. Native variance is less than 1%, 2. Java variance is less than 1%
Roadmap id: CARD-134
Work Items
Work items:
Work with the TCWG to reduce variance: DONE
Integrate suggestions: DONE
Integrate into benchmarking framework: DONE
Document improvements: DONE
Get sign-off: DONE
