Android Toolchain Benchmarking

Registered by Andy Doan on 2011-10-13

Give an overview of what is currently being done for Android toolchain benchmarking in Linaro. Discuss future enhancements we should look into.

Agenda:
 * explain what I currently do each month
 * discuss some of the short-comings
 * discuss some of the future ideas we have and get other ideas on what we should do to improve this

Goals:
 * decide what areas we should focus on related to android toolchain benchmarking going forward
 * get an idea of the effort required
 * does Andy have the time to do this on his own

Blueprint information

Status:
Complete
Approver:
Andy Doan
Priority:
Essential
Drafter:
Andy Doan
Direction:
Approved
Assignee:
Andy Doan
Definition:
Approved
Series goal:
Accepted for 2011q4
Implementation:
Informational Informational
Milestone target:
milestone icon connect-q4.11
Started by
Zach Pfeffer on 2012-02-27
Completed by
Zach Pfeffer on 2012-02-27

Related branches

Sprints

Whiteboard

More background can be found at:
  https://wiki.linaro.org/Platform/Android/AndroidToolchainBenchmarking

Two topics we should discuss are:

 1) https://blueprints.launchpad.net/linaro-android/+spec/android-0xbench-lava
 2) https://blueprints.launchpad.net/linaro-android/+spec/lavatize-toolchain-benchmarking

[zpfeffer 2011/10/31]
Possible action items:
    Fix 0xbench so it runs both manually and automated
    Integrate 0xbench and Andy's flow for both manual and automated runtime
    Integrate GLmark2 and get FPS
    Get SMP benchmarks/test
    Integrate CTS
More benchmarks (from Tony)
    SunSpider
    V8
    Vellamo
    IE Fishtank (http://ie.microsoft.com/testdrive/performance/fishietank/)
    JSGamebench (http://qc.no.de/)
See https://blueprints.launchpad.net/linaro-android/+spec/linaro-platforms-lc4.11-android-qa

Notes from LDS
Notes

https://wiki.linaro.org/Platform/Android/AndroidToolchainBenchmarking/2011-10
is generated by a script.

AI: <Android/Infrastructure>: Create cohesive benchmark and benchmark results saving

AI: Andy: Need to integrate Andy's flow into LAVA
  Need to scope
     Need to compile on host
     ADB
     Cross installed (will need to be able to install cross toolchain)
  Do we do this as part of the toolchain build?
  Test n times each
  Frans to help
  Need to handle new options
    Need a config option
  Need to run manually/LAVA
  Need to be able to auto gen report current vs flot

What happens when we have a regression?
  Month-to-month
  Image-to-image
  What resources do we have to investigate
  Toolchain group can get Skia expert?
  perf, 1st level
    assembler
    gen asm diff of hotspot,
  Can toolchain start to use Android's workloads

Look at the history we have

AI: Need to strip down Android image and benchmark
   Need way to stop services

Benchmark Bionic?
Just Benchmark Skia
Need to surgicially target specific libs
AI: Alexander to work with toolchain group to help them sort out a basic Android test rig
    SSH in
    Remote access
AI: LAVA needs to keep an image live
AI: <Android>: Set up remote acess
    Need sshd

AI: Andy to sync up with toolchain group about the agreed on benchmark set

AI: SMP?

benchmark.git, cx image change

AI: <Android>: Reach out to benchmark.git

AI: Need LAVA to support parameters in, Yong Qin

(?)

Work Items