Touch Core Apps testing needs

Registered by Nicholas Skaggs on 2013-11-06

With the migration to 1.4 and trusty, the core apps workflow for testing and merging into the core images still has some pain points that need addressing.

The core apps tests in many cases don't realibly run in a testing enviorment. There are several bad implications from this:

-Merge request by developers fail because a test randomly fails for code they didn't change
-Stable images can't be produced or are produced with failing tests
-True bugs that the tests find are hard to seperate from the noise (is this a random failure, or a bug?)
-Once a bug is found, the cause of the bug is hard to diagnose (platform/sdk introduced, core apps code change, autopilot, ci infastructure, etc)

Let's talk about getting the core apps tests green and running reliably. In addition, let's talk about workflow changes to revert code when the tests break in order to keep the app in good shape.

Blueprint information

Status:
Not started
Approver:
Jono Bacon
Priority:
Undefined
Drafter:
Nicholas Skaggs
Direction:
Approved
Assignee:
Nicholas Skaggs
Definition:
New
Series goal:
Accepted for trusty
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

We talked a bit about how to clean up and fix the tests in this session;
http://summit.ubuntu.com/uds-1311/meeting/22071/appdev-1311-core-apps-test-review/

We didn't get to do as much indepth review. It might be useful to do so now in this session and ensure all the issues have folks assigned with work items.

what if the api changes?
    <mzanetti> ideally 2 steps. mock the service, test app with qmltestrunner. have autopilot tests walking through the api
    <cjohnston> using an external serivce for testing can provide you with false failures
    <mzanetti> 2-step testing... 1) mocks + unit tests, 2) integration tests that that it the api still works
    <mzanetti> yeah... you don't need too many AP tests then. Only a few that interact with the server.
    how will we know what the failure was caused
        remote server testsuite shouldn't block CI
            hard to get value out of running tests
                perhaps leave an API change like that to manual tests
    Rely on user reports of failures, use mock api's for testing

COPIED FROM PREVIOUS SESSION :-)

ubuntu-weather-app-autopilot 2013/11/20 15:47 17 14 3 0 0 82.4%

    Leo wants to refactor

    remember to mockup for network issues

    2 tests need to be re-enabled and tested (swipe to delete emulator)

    test_locationmanager.TestLocationManagerWithLocation.test_cancel_remove_location

    test_locationmanager.TestLocationManagerWithLocation.test_remove_location

    missing tests

    test to swipe for different hours in the day

ubuntu-terminal-app-autopilot 2013/11/20 15:39 9 9 0 0 0 100%

    all good :-)

ubuntu-rssreader-app-autopilot 2013/11/20 15:32 5 2 3 0 0 40%

    forces us to use internet resources which can cause failures

    https://bugs.launchpad.net/ubuntu-rssreader-app/+bug/1248737

    needs smaller tests, a bit of refactoring

ubuntu-filemanager-app-autopilot 2013/11/20 15:17 25 8 17 0 0 32%

    remove patching home

    refactor to use new folder

ubuntu-clock-app-autopilot 2013/11/20 15:10 11 11 0 0 0 100%

    refactor to only use local sources

    land mp's

    https://code.launchpad.net/~nik90/ubuntu-clock-app/more-stopwatch-tests/+merge/193922

    alarm tests

    UI alarm test in autopilot

    lower level tests for push notifications

ubuntu-calculator-app-autopilot 2013/11/20 14:59 30 30 0 0 0 100%

    some tests better as unit tests, but they all work and features covered

dropping-letters-app-autopilot 2013/11/20 14:55 5 5 0 0 0 100%

    Good, no code changes, tests rock solid

music-app-autopilot 2013/11/20 14:48 8 7 1 0 0 87.5%

    shuffle implementation and tests have been an issue

    https://bugs.launchpad.net/music-app/+bug/1253480

    Needs tests for interfacing with Aritsts, Albums, Songs tabs

    https://bugs.launchpad.net/music-app/+bug/1253453

    https://bugs.launchpad.net/music-app/+bug/1253452

    https://bugs.launchpad.net/music-app/+bug/1253449

    https://bugs.launchpad.net/music-app/+bug/1253447

    Needs tests for each tab to add an item to a playlist or the queue

    https://bugs.launchpad.net/music-app/+bug/1253454

    Needs tests to create and modify playlists

    https://bugs.launchpad.net/music-app/+bug/1253457

    Needs tests for swipe to delete of playlist items and queue items

    https://bugs.launchpad.net/music-app/+bug/1253697

calendar-app-autopilot 2013/11/20 14:41 7 2 5 0 0 28.6%

    3 mp's need to land / merge.

    once landed all features should have tests

    https://code.launchpad.net/~acerisara/ubuntu-calendar-app/failing-tests/+merge/193706

    https://code.launchpad.net/~vthompson/ubuntu-calendar-app/merge-ap-fix-branches has the AP 1.4 fixes as well as a few others. It is green as far as Jenkins can see. I think balloons has a merge that will fix a test that's failing on the device.

ubuntu-docviewer-app

    status?

    what's development look like?

sudokutouchgame

    not in public jenkins view

    can we add?

    toolbar changed and broke the tests

stock-ticker-mobile-app

    not in public jenkins view

    can we add?

    https://code.launchpad.net/~veebers/stock-ticker-mobile-app/porting-autopilot-1.4/+merge/193718
    https://code.launchpad.net/~veebers/stock-ticker-mobile-app/mock-server-for-testing/+merge/194971

(?)

Work Items

Work items:
[nskaggs] weather app, 2 tests need to be re-enabled and tested (swipe to delete nskaggs): DONE
[nskaggs] add bug for weather app, test to swipe for different hours in the day: DONE
[nskaggs] filemanager, refactor to use new folder: DONE
[nik90] clock, refactor to only use local sources: TODO
[nskaggs] msuic, remove shuffle test: DONE
[vthompson] music, look into shuffle test issues and fix: DONE
music app has lots of new tests to be written: DONE
[nskaggs] calendar, compare victor's mp, merge victor's mp, and potentially merge mine as well: DONE
[chris.gagnon] fix sudokutouch; toolbar issues, et la: TODO
[nskaggs] open bug to add sudokutouch and stock-ticker to the daily image testing, https://bugs.launchpad.net/ubuntu-ci-services-itself: DONE

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.