Investigate Android Benchmark Variance

Registered by Zach Pfeffer on 2012-08-28

Why?
The Toolchain Working Group has requested that we create a system where the run-to-run variance of our benchmarks is 1% for native and JAVA workloads. These low -variances will allow comparisons across baselines and allow benchmark guided optimization. An effort t olower this variance was done and the feedback from the TCWG was to look at the higher variances and see what's causing them.

Context?
This is part of the larger ARM benchmarking effort.

What gets produced?
A mechanism that runs before a benchmark that puts the system into a lower variance state han it was or a list of benchmarks who's variances cannot be characterized or lowered.

Where will the work get put?
A script on android.git.linaro.org that gets included in the build.

Blueprint information

Status:
Complete
Approver:
Zach Pfeffer
Priority:
Medium
Drafter:
vishal
Direction:
Approved
Assignee:
vishal
Definition:
New
Series goal:
Accepted for 2012q3
Implementation:
Implemented
Milestone target:
milestone icon 12.09
Started by
Zach Pfeffer on 2012-09-04
Completed by
Zach Pfeffer on 2012-09-04

Related branches

Sprints

Whiteboard

Notes:
[2012/8/22 pfefferz] Please put notes here.
[2012/9/04 vishalbhoj] inputs from TCWG wrt signoff:

Benchmarks are consistent,with lower variance and we have explanations for higher variance .

Some of those with high variance were due to
 * GC
 * Database calls causing I/O
 * Stream copy and stdlib copy

The A9 has higher variance on memory and branch predictor than the A8, should try on a BeagleBoard.
try memory benchmarks on cortex-A8 board (beagleboard).

Meta:
Headline: Native and JAVA benchmarks can be run with 1% variance and we can use benchmarks to optimize
Acceptance: 1. Native variance is less than 1%, 2. Java variance is less than 1%
Roadmap id: CARD-134

(?)

Work Items

Work items:
Work with the TCWG to reduce variance: DONE
Integrate suggestions: DONE
Integrate into benchmarking framework: DONE
Document improvements: DONE
Get sign-off: DONE

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.