Ubuntu Touch Image Testing

Registered by Paul Larson on 2013-04-12

Setup daily smoke testing for ubuntu touch images on the devices

Blueprint information

Status:
Not started
Approver:
Gema Gomez
Priority:
Essential
Drafter:
Paul Larson
Direction:
Approved
Assignee:
Canonical CI Engineering
Definition:
Approved
Series goal:
Accepted for saucy
Implementation:
Unknown
Milestone target:
milestone icon ubuntu-13.10

Related branches

Sprints

Whiteboard

UDS Discussion
------------------------

Situation today:
* Lack of planning, adhoc testing -> Creation of a test plan for functional and non-functional system wide testing
* Lack of smoke testing for utuntu touch images

* Autopilot 1.3 is pending it's release, a phablet autopilot is used today but we halted development for it; some tests are broken due to this.

Short term plan:
* Put smoke testing in place ASAP
* Have cdimage pull bits into /pending.
* UE-jenkins to trigger copy from /pending to /current
* app-specific autopilot tests still belong in PS
* UE will investigate PS autopilot tests for suitability to use for smoke testing or come up with additional smoke tests

Interesting non-UI tests that would be valuable?
- Identify if the hardware bits are working (camera, GPS, wifi, etc)
- Indicators are valuable and worth testing during smoke
- ogra has code that queries sanity of the image already there
  - actually he looks at sergio's results of application autopilot tests

asac's post session comments (on test targets vs. x86 vs. emulator)
 + yes, we want to test on real devices
 + yes, we want to support our desktop users
 + yes, we want to use emulator, even for x86 for that
 + x86 image will be required, but android upstream and goldfish support that, so phone/foundation should be able to do this.
 + Q for QA: can you run goldfish emulator?
 [plars] once it's supported by the images, we would treat it just as another device type, I don't foresee any problem to support this

asac's post session comments/braindump on what smoke tests:
 + check if hardware is available through /proc or other basic stuff like ifconfig etc.
 + check out the android testsuites available in AOSP for the hal layer - might require dalvik to use most of those, but some tests are available in C
 + add all core smoke tests you would usually run on ubuntu?
 + add something that uses the whole UI
 + pull in tests developed by the components developers (e.g. our camera application tests etc.)
 + agreed that you cannot skip UI; just start testing low level basics and then add something high level using full stack as well.
 + I am not sure that defining smoke testing from QA perspective (the definition saying "fast fail to determine if more testing is OK") is the right approach. I believe smoke testing should reveal all the critical tests that prevent an image to become available to the inner ring of users/stakeholders: developers, hardcore users/designers, managers and sales.
 + smoke tests are those tests that were decided to hold the presses and immediately refocus all energy leaded to fix these - potentially accepting delay on all regular projects.
 + consider: asac has the mandate to call people at any point of time and get them drop all and fix the smoke tests. I believe we want to do that for strategic issues as well as issues that block a good proportion of inner circle users from above

Session Summary
---------------------------
Interesting non-UI tests that would be valuable?
- Identify if the hardware bits are working (camera, GPS, wifi, etc)
- Indicators are valuable and worth testing during smoke
- images are currently tested in PS using the application autopilot tests
asac's post session comments/braindump on what smoke tests:
 + check if hardware is available through /proc or other basic stuff like ifconfig etc.
 + check out the android testsuites available in AOSP for the hal layer - might require dalvik to use most of those, but some tests are available in C
 + add all core smoke tests you would usually run on ubuntu?
 + add something that uses the whole UI
 + pull in tests developed by the components developers (e.g. our camera application tests etc.)
 + agreed that you cannot skip UI; just start testing low level basics and then add something high level using full stack as well.
 + I am not sure that defining smoke testing from QA perspective (the definition saying "fast fail to determine if more testing is OK") is the right approach. I believe smoke testing should reveal all the critical tests that prevent an image to become available to the inner ring of users/stakeholders: developers, hardcore users/designers, managers and sales.
 + smoke tests are those tests that were decided to hold the presses and immediately refocus all energy leaded to fix these - potentially accepting delay on all regular projects.
 + consider: asac has the mandate to call people at any point of time and get them drop all and fix the smoke tests. I believe we want to do that for strategic issues as well as issues that block a good proportion of inner circle users from above

touch testing:

- default: run full settle script
- before every test: run it with 30 second per measurement
- after test just run just one measurement (e.g. one time 30 seconds)
99.00

(?)

Work Items

Work items:
[pwlars] investigate how to run autopilot tests on touch: DONE
[doanac] incoroporating lttng run data into a job for touch images: TODO
[gema] Create a test plan for non-functional testing: DONE
[gema] Create a utah runlist for basic smoke tests on the phablet image: DONE
[gema] Move the utah basic smoke tests for phablet to production: DONE
[gema] Make sure we move to run the new test cases on the smoke testing for phablet jobs: DONE
[gema] Bootstrap the functional testing of ubuntu touch: DONE
[gema] Work with design on non-functional requirements: DONE

Work items for ubuntu-13.05:
[gema] Review thomir's proposal for benchmarking format and send feedback: DONE
[pwlars] convert smem jobs on mobile from quantal to raring: DONE
[pwlars] setup boot test job (will eventually hold smoke tests) for mako: DONE
[pwlars] setup boot test job (will eventually hold smoke tests) for maguro: DONE
[pwlars] setup boot test job (will eventually hold smoke tests) for grouper: DONE
[gema] Figure out how to run things on the phablet images, document it for others to use: DONE
[gema] Figure out what information needs to be extracted from the images installed on the mobiles for traceability: DONE

Work items for ubuntu-13.06:
[doanac] Get UTAH ready to gather image information from the touch images: DONE
[pwlars] get eventstat jobs running again on new phablet images: DONE
[pwlars] setup boot test job (will eventually hold smoke tests) for manta: DONE
[pwlars] fix issues with phablet install/setup/test scripts: DONE
[pwlars] move smem testing on touch to utah: DONE
[pwlars] setup eventstat testing on touch using utah: DONE
[pwlars] create scripts and templates to automatically regenerate jenkins jobs for touch smoke and nonfunctional tests: DONE
[pwlars] extend boot tests to include smoke testing: DONE
[om26er] create new autopilot package for phone-app to simulate calling: DONE
[gema] Work with the sdk team to get some image test cases in place to be able to run daily (initially we will rely on app testing that is already in autopilot, to be run on daily images): DONE

Work items for ubuntu-13.07:
[gema] Monitor the addition of autopilot basic test cases to the phablet utah runlist and run autopilot tests from utah: DONE
[gema] Gather information and come up with a plan on how to approach ui toolkit testing (https://wiki.ubuntu.com/QATeam/AutomatedTesting/UIToolkit): DONE
[doanac] setup jenkins to run touch tests for all the autopilot apps: DONE
[pwlars] create parameterized mark-pending-current job for touch images: DONE
[sergiusens] Identify some non-UI related tests that could be added for smoke testing: DONE

Work items for ubuntu-13.08:
[doanac] Get power-idle working on touch images: POSTPONED
[gema] Coordinate lab set up and test case development for connected tests: DONE
[saviq] Get autopilot tests fixed for touch so that they can be added to smoke testing in the lab next week (20 May 2013): DONE
[ogra] Make sure the plumbing for prepublishing smoke testing of ubuntu touch images is in place for smoke testing to be able to run before publishing: TODO
[sergiusens] Make sure the plumbing for prepublishing smoke testing of ubuntu touch images is in place for smoke testing to be able to run before publishing: TODO
[pwlars] restructure touch smoke jobs for improved lock safety and serialization: DONE
[sergiusens] Create hybris runnable (utah compatible) tests: INPROGRESS
[sergiusens] Create platform-api runnable (utah compatible) tests: TODO
[sergiusens] Create ofono runnable (utah compatible) tests: TODO
[sergiusens] Create powerd runnable (utah compatible) tests: TODO
[pitti] create apport script to send all pending crash reports to errors.u.c. (but not to LP): DONE
[pwlars] call /usr/share/apport/whoopsie-upload-all to send crash reports at the end of test runs: DONE
[om26er] create jenkins jobs to run autopilot tests on devices: DONE
[doanac] Utah need to support memory restrictions on touch images (phablet-flash --cmdline mem=386m): BLOCKED
[pwlars] Add smoke testing touch jobs with memory restrictions: TODO

This blueprint contains Public information 
Everyone can see this information.