Testing with the ubuntu sdk

Registered by Nicholas Skaggs on 2013-08-15

As an application developer targetting the ubuntu touch platform, testing plays a key role in your development needs. Let's talk about what tools are availible to you as an application developer and how you can use them to help deliver a well tested application. In addition, let's gather feedback on your needs, and if they are being met.

Blueprint information

Status:
Not started
Approver:
Jono Bacon
Priority:
Undefined
Drafter:
None
Direction:
Needs approval
Assignee:
None
Definition:
New
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

Agenda:

Testing needs:
unit testing
Functional / end user testing

Comments on tools
Qt Creator
Autopilot
Ubuntu SDK Emulator
lp:ubuntu-ui-toolkit
as sdk updates, we can update the module and not be forced to update each individual test
Work in progress, but is being used no

Qml Testing
better suited to unit testing
much faster
no need to start full application or dbus
can easily verify application behaves as you expect
Examples:
lp:unity8
lp:ubuntu-sdk-tutorials
Anything that is self-contained makes sense for qml
should run everywhere qt can run
supports c++ compiled code, etc
qml uses the toolkit directly to automate, while autopilot uses the dispay server, signals, etc
changes to sdk potentially have a more limited impact
emulators less needed
less shared code
more specific tests
unity8 has collection of helper functions
could be upstreamed
http://bazaar.launchpad.net/~unity-team/unity8/trunk/view/head:/tests/utils/modules/Unity/Test/UnityTestCase.qml

Feedback:

We need a way to tell when a test should be written in autopilot, and when it should be written in qml or c.
discuss about this after the training session

We need a suite for the sdk user-facing features. What we currently have is a suite for the emulators and a suite for the showcase.
can we review coverage of sdk?
no tests that are interactive

For the tests written in autopilot, we need a better way to tell if we are testing the installed modules or the modules from source code.
if you have installed modules as well as the source code, can be confusing and caused incorrect results
testrunner could run "testing" version from upstream source
hardcoded paths don't seem to be the answer
clean qmlscene code found in __init__.py

We need to add a base test case to ubuntu-ui-toolkit that's inherited by all the core app suites. That way we will be able to change things in the base test case, instead of changing them in all the apps.

 we need templates/better templates in qtcreator for everything
 autopilot test template
 qml testing tempate?
 not worth implementing?
 autopilot test with emulator template? We need first the base test case.

the ubuntu sdk emulator for autopilot could include verbose logging for each action and call

(?)

Work Items

Work items:
[nskaggs] Update the QML testing tutorial: TODO
[nskaggs] IRC training session for QML testing (coordinate as part of showdown?): TODO
[nskaggs] create qml testing video: TODO
[nskaggs] get with ubuntu sdk team (juhapekka?) to review sdk testing coverage; are all components well covered? [juhapekka maybe?]: TODO
[elopio] clean up qmlscene installed vs local checks: TODO
[elopio] write a base test case to be used by all the apps: TODO
[mzanetti] expand base template with simple qml test: TODO
[elopio] propose an annotation for logging emulator actions.: TODO

This blueprint contains Public information 
Everyone can see this information.