Make a list of tests present in Jellybean AOSP source

Registered by Zach Pfeffer on 2012-07-27

Why?
We want to track which tests are in present in AOSP Jellybean. To make that possible we need to construct (and possibly maintain over time) a list of tests that are present in particular git repositories.

Context?:
The testing card.

What gets produced?
1) The scripts that compute this as well as the actual data.

Where will the work get put?
1) https://github.com/zyga/android-test-tool
2) A new Google spreadsheet (https://docs.google.com/a/linaro.org/spreadsheet/ccc?key=0AgiculnRh94adHJacmtWckFDVjVlcXdseXBISG1GRlE)

Blueprint information

Status:
Complete
Approver:
Zach Pfeffer
Priority:
High
Drafter:
Zygmunt Krynicki
Direction:
Approved
Assignee:
Zygmunt Krynicki
Definition:
Approved
Series goal:
Accepted for 2012q2
Implementation:
Implemented
Milestone target:
milestone icon 12.08
Started by
Zygmunt Krynicki on 2012-07-27
Completed by
Zach Pfeffer on 2012-09-05

Whiteboard

Notes:
[2012-07-26 pfefferz] Put notes here please.
[2012-07-27 zkrynicki] Built JB for galaxy nexus with and without tests. Running diff tools I wrote earlier.
[2012-07-30 zkrynicki] Had a great weekend. Now off to work: I analyzed my JB builds for nexus and did a few diffs. I also wrote a small helper to compare ICS and JB easily. I wrote a summary of my research to the mailing list: http://lists.linaro.org/pipermail/linaro-android/2012-July/001378.html I'm going to start reading the source code of the existing tests in sequence to try to discover the useful bits 'dry' (I don't have any way of running JB as I don't have a nexus device).
[2012-07-30 zkrynicki] Working on a list of tests that also includes startup instructions and some analysis https://docs.google.com/a/linaro.org/spreadsheet/ccc?key=0AgiculnRh94adHJacmtWckFDVjVlcXdseXBISG1GRlE
[2012-08-01 zkrynicki] Refocusing on panda builds and gtest tests as I don't think test APKs are worthwhile now
[2012-08-01 zkrynicki] 'tests' variant of panda+jelly bean _does not work_ (reboot loop after initial installation of all the apks), wasted day trying to debug and analyze, notified bhoj for assistance. UPDATE: 'eng' build fails the same way, just faster. I suspect that running disablesuspend.sh 'damages' the system in some way. Testing to see what happens when I never run disablesuspend.sh. UPDATE: confirmed, disablesuspend.sh breaks panda JB. Reported as bug 1031924
[2012-08-01 zkrynicki] Booting 'tests' build gets the board stuck in a boot-loop.
[2012-08-03 zkrynicki] Aborting attempts to use JB, reverting to ICS before stability improves to usable level.
[2012-08-07 zkrynicki] Added a wiki page: https://wiki.linaro.org/Platform/Android/TestsInIcsAndJb
[2012-08-07 zkrynicki] Working on a test script using ICS as a baseline, it should work without changes on JB.
[2012-08-08 zkrynicki] Created C wrapper that runs gtests and creates lava test bundles. Zach has created a new rpository in gerrit for landing procedures.
[2012-08-09 zkrynicki] I'm moving to JB builds as they had more tests I wanted to run (multi-test in one executable)
[2012-08-10 zkrynicki] More work on the C wrapper.
[2012-08-13 zkrynicki] Discussed the C wrapper with liuyq, laid out plans to integrate with lava-android-test as a test provider or parametrized test. Added support for additional JB features that were added to gtest framework (multi-test runs), fixed a few minor bugs and preparing to get it out the door to gerrit.
[2012-08-13 zkrynicki] Reported a bug that can cause problems if we want to roll out LAVA integration. Those bugs need workarounds in LAVA or fixes in our build tools. They can only be fixed with serial console running as root on the device and cannot be fixed over adb. https://bugs.launchpad.net/linaro-android/+bug/1036365
[2012-08-14 zkrynicki] Bundle from surface flinger tests that ran on my snowball: http://validation.linaro.org/lava-server/dashboard/streams/anonymous/zyga/bundles/acba9cf336b14da328de0e43d0b1b537b7f1995f/
[2012-08-14 zkrynicki] Pushed working code to https://github.com/zyga/android-lava-wrapper - the code needs rebasing to create proper commits. It also needs README.md with accurate usage instructions.
[2012-08-20 zkrynicki] Back from holiday!
[2012-08-20 zkrynicki] Uploaded some results from my current work today: http://validation.linaro.org/lava-server/dashboard/streams/anonymous/zyga/bundles/ One of the result failed to deserialize, I've fixed the problem locally. Code will land in github once my builds finish.
[2012-08-21 zkrynicki] Fully working wrapper landed in github in the develop branch. I will merge it to master branch, squash history and integrate that into our builds.
[2012-08-22 zkrynicki] Working on lava-android-test support. Got stuck on adb shell running as 'shell' user in -tests builds. Tracing how to resolve that. I want to run adb shell as root in such builds. Bhoj suggested that I look at system/core/adb/adb.c + 898 and I believe everything dependson ALLOW_ADBD_ROOT ifdef flag. Fixed locally and pushed to gerrit: http://review.android.git.linaro.org/#change,2671
[2012-08-23 zkrynicki] Clean build with the patch above work as expected, it was just a partial build issue - big relief :-)
[2012-08-24 zkrynicki] Renamed lava-wrapper to lava-blackbox to better reflect what it does.
[2012-08-24 zkrynicki] Implemented lava-android-test integration. Everything works as expected but I want to change how run & parse cooperate before proposing this. Sample result in http://validation.linaro.org/lava-server/dashboard/streams/anonymous/zyga/bundles/a2e3b9a9ff3181152c7e5ac074162e88fccc2eeb/ code in lp:~zkrynicki/lava-android-test/blackbox
[2012-08-25 zkrynicki] Proposed l-a-t integration for review for comments: https://code.launchpad.net/~zkrynicki/lava-android-test/blackbox/+merge/121299
[2012-08-27 zkrynicki] Discussing the merge proposal above with liuyq
[2012-08-27 zkrynicki] Spun off the integration bits to dedicated blueprint: https://blueprints.launchpad.net/linaro-android/+spec/automate-in-tree-aosp-test
[2012-09-05 pfefferz] There was one undone WI, Investigate how to run each test: POSTPONED. This WI has been removed since the follow on work in https://blueprints.launchpad.net/linaro-android/+spec/automate-in-tree-aosp-test runs the tests.

Meta:
Roadmap id: ANDROID2012-AOSP-TESTS
Headline: All the tests in Jellybean have been examined
Acceptance: All the tests in Jellybean have been examined

(?)

Work Items

Work items:
Extract all tests from JB build: DONE
Generate list: DONE
Put summary on the wiki: DONE

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.