Automate In-Tree AOSP Tests

Registered by Zach Pfeffer on 2012-08-27

Why?
We have our list of tests and we need to create a mechanism to automate them.

Context?:
The testing card, "Automate AOSP Test Cases": http://cards.linaro.org/browse/CARD-146

What gets produced?
A mechanism to run the tests available in the AOSP source.

Where will the work get put?
1) https://github.com/zyga/android-test-tool
2) A new script to run those tests https://github.com/zyga/android-lava-wrapper

Blueprint information

Status:
Complete
Approver:
Zach Pfeffer
Priority:
Low
Drafter:
Zygmunt Krynicki
Direction:
Approved
Assignee:
Zygmunt Krynicki
Definition:
Approved
Series goal:
Accepted for 2012q2
Implementation:
Implemented
Milestone target:
milestone icon 12.09
Started by
Zygmunt Krynicki on 2012-08-29
Completed by
Zach Pfeffer on 2012-09-24

Related branches

Sprints

Whiteboard

Notes:
[2012-08-27 pfefferz] Put notes here please.
[2012-08-27 pfefferz] Split from https://blueprints.launchpad.net/linaro-android/+spec/aosp-tests-on-jb due to scope realignment
[2012-08-28 zkrynicki] I've encountered a blocker in lava-android-test that prevents saving of data. In my opinion this cannot be fixed properly without changing the (IMHO broken) API that test definitions can override. The core of the problem is that something as simple as directly offering the full bundle from the test definition is impossible. The parts that are not customizable, specifically the generate_bundle() function, ultimately control what happens. I've discussed this with liuyq and we're looking for a solution that would not require large changes.
[2012-08-28 zkrynicki] The solution proposed by liuyq was not acceptable as it would require adding a new top-level command to lava-android-test just to run blackbox. I've chosen to monkey patch the framework after considering all other alternatives. The merge proposal is now up: https://code.launchpad.net/~zkrynicki/lava-android-test/blackbox/+merge/121299
[2012-08-29 zkrynicki] Marking as approved. Marking implementation as deployment as my role now is to ask people for help with landing this.
[2012-08-29 zkrynicki] Landed to trunk lp:lava-android-test. Andy is pushing this to staging. I'll run a few jobs with this test using my own custom build for panda.
[2012-08-30 zkrynicki] Renamed lava-android-wrapper to lava-blackbox (https://github.com/zyga/lava-blackbox). Merged develop to trunk. Running final tests on panda (I was working on snowball previously)
[2012-09-05 pfefferz] Bump to 12.09, good progress was made in 12.08, but there are still a few things to finish.
[2012-10013 zkrynicki] lava-blackbox has landed to all JB builds. Test builds worked on various machines and fast models. I've asked dave or andy to deploy release current lava-android-test and deploy it to production. We should have lava-blackbox in daily tests shortly.

Meta:
Roadmap id: ANDROID2012-AOSP-TESTS
Headline: Many tests available in the AOSP source have been automated
Acceptance: 1. Many tests available in the AOSP source have been automated
2. The mechanism that runs the tests has been made available and has been documented

(?)

Work Items

Work items:
Write a wrapper for google test tests: DONE
Package lava-wrapper properly with Android.mk: DONE
Fix adb shell to run as root in -tests builds: DONE
Integrage lava-wrapper into our manifests: INPROGRESS
Prepare to integrate into LAVA: DONE
Review lava-android-test integration: DONE
Land blackbox support to lava-android-test: DONE
Rename android-lava-wrapper to lava-blackbox: DONE
Patch one build config to run the blackbox tests: INPROGRESS
[liuyq0307] Make build with http://paste.ubuntu.com/1187127/: DONE
[liuyq0307] Add lava-blackbox to builds after its been pushed to production: INPROGRESS
[zyga] Push a build to staging to test (once 1 is done): DONE
http://review.android.git.linaro.org/2671 -- could be pushed to google too: DONE
[zyga] Ask Andy to push to production: INPROGRESS
[zyga] Push everything to Gerrit as one patch: TODO
[liuyq0307] Push manifest change to http://paste.ubuntu.com/1187127/: DONE

This blueprint contains Public information 
Everyone can see this information.