Ubuntu Touch event-based testing

https://docs.google.com/a/canonical.com/document/d/1XjgLa59wFhdqErXb4rHs3FoojsDbMyf0oQa0Km2W-qY/edit

Goals [13.06]: Memory measurements are taken during/after test case execution with autopilot against images and reported to staging QA dashboard
Goals [13.07]: Memory measurements are integrated with UTAH and exported to the QA production dashboard. Budgetting for system/app memory usage is put into place.

Work items for ubuntu-13.06:
[thomas-voss] Define which events need to be measured: DONE
[doanac] Come up with the architecture to be able to run smem at the right time: DONE
[javier.collado] Come up with the architecture to be able to run smem at the right time: DONE
[javier.collado] Investigate tools to get kernel events (lttng, SystemTap, proc connector): DONE
[javier.collado] Setup development environment in a raring desktop image: DONE
[javier.collado] Review autopilot test cases already written for each of the apps: DONE
[javier.collado] Write autopilot tests that trigger each requested event one by one and measures memory usage for the application involved: DONE
[javier.collado] Investigate what's the best approach to run multiple applications in a single autopilot test case: DONE
[javier.collado] Merge test cases in a single big scenario that triggers all the events in a sequence and measures memory usage for all the applications involved: DONE
[javier.collado] Package test scenario to run it easily with phablet-test-run: DONE
[javier.collado] Receive testing device: DONE
[javier.collado] Validate autopilot test in a real device with a touch image installed: DONE
[javier.collado] Gather preliminary data and get feedback about its usefulness: DONE

Work items for ubuntu-13.07:
[javier.collado] Write UTAH runlist that wraps the autopilot test case: DONE
[javier.collado] Launch UTAH runlist locally using the run_utah_phablet.py script: DONE
[javier.collado] Create a jenkins job that runs the UTAH runlist locally: DONE
[javier.collado] Create setup script and template to create/update job in a give server: DONE
[javier.collado] Agree with QA dashboard developers on the format of the results: DONE
[javier.collado] Deploy UTAH runlist in the lab using a jenkins job to run it daily on touch images with real devices: DONE
[javier.collado] Creation of new views in the QA dashboard for reporting these test results: TODO

Work items for ubuntu-13.08:
[larry-e-works] Set stable content in the lab for the web browser: DONE
[joetalbott] Add flag on whether memory tests passed or not to show memory events tests as passing or not: TODO
[joetalbott] Move memevents tests to production: DONE
[javier.collado] Add threshold values and results to the measurements json file: DONE
[javier.collado] Set job return code depending on the threshold results: DONE
[joetalbott] Flag tests as failing in the dashboard when above defined thresholds: INPROGRESS
[javier.collado] Limit memory on the runs to TODO MBs - (not supported by phablet-flash): BLOCKED
[javier.collado] Add option to discard values from testing processes (UTAH & autopilot) - (smem -P filters output, but PSS measurement won't change because of the filtering): BLOCKED
[javier.collado] Try to capture shell_started and shell_unlock events: POSTPONED
[javier.collado] Track jenkins job results and ensure test is stable: DONE
[javier.collado] Review measurement architecture to check if it can be reused for other tests: POSTPONED