Create a mechanism to create a low-variance environment for Android benchmarks

Registered by Zach Pfeffer on 2012-07-03

Why?
The Toolchain Working Group has requested that we create a system where the run-to-run variance of our benchmarks is 1% for native and JAVA workloads. These low -variances will allow comparisons across baselines and allow benchmark guided optimization.

Context?
This is part of the larger ARM benchmarking effort.

What gets produced?
A mechanism that runs before a benchmark that puts the system into a low variance state.

Where will the work get put?
A script on android.git.linaro.org that gets included in the build.

Blueprint information

Status:
Complete
Approver:
Zach Pfeffer
Priority:
Medium
Drafter:
vishal
Direction:
Approved
Assignee:
vishal
Definition:
Approved
Series goal:
Accepted for 2012q2
Implementation:
Implemented
Milestone target:
milestone icon 12.08
Started by
vishal on 2012-07-23
Completed by
Zach Pfeffer on 2012-08-28

Related branches

Sprints

Whiteboard

Notes:
[2012/7/2 pfefferz] Please put notes here.
[2012/7/23 vishalbhoj] We have deployed benchmarking applications into LAVA for snowball and pandaboard. We are observing around variance of less than 1 percent in most of the native tests. Here are the results for pandaboard:
https://docs.google.com/a/linaro.org/spreadsheet/ccc?key=0AluJE1IQS-rPdGswX0psR1VIRkRKdTczRHdtNURLYUE&pli=1#gid=0
[2012/7/23 pfefferz] Moving to 12.08. Other Benchmarking tasks took longer than expected.

Meta:
Headline: Native and JAVA benchmarks can be run with 1% variance.
Acceptance: 1. Native variance is less than 1%, 2. Java variance is less than 1%
Roadmap id: CARD-134

(?)

Work Items

Work items:
Create script: DONE
Measure native variance: DONE
Measure JAVA variance: DONE
Get feedback from the TCWG: DONE

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.