Touch Core Apps testing needs
With the migration to 1.4 and trusty, the core apps workflow for testing and merging into the core images still has some pain points that need addressing.
The core apps tests in many cases don't realibly run in a testing enviorment. There are several bad implications from this:
-Merge request by developers fail because a test randomly fails for code they didn't change
-Stable images can't be produced or are produced with failing tests
-True bugs that the tests find are hard to seperate from the noise (is this a random failure, or a bug?)
-Once a bug is found, the cause of the bug is hard to diagnose (platform/sdk introduced, core apps code change, autopilot, ci infastructure, etc)
Let's talk about getting the core apps tests green and running reliably. In addition, let's talk about workflow changes to revert code when the tests break in order to keep the app in good shape.
Blueprint information
- Status:
- Not started
- Approver:
- Jono Bacon
- Priority:
- Undefined
- Drafter:
- Nicholas Skaggs
- Direction:
- Approved
- Assignee:
- Nicholas Skaggs
- Definition:
- New
- Series goal:
- Accepted for trusty
- Implementation:
-
Unknown
- Milestone target:
- None
- Started by
- Completed by
Whiteboard
We talked a bit about how to clean up and fix the tests in this session;
http://
We didn't get to do as much indepth review. It might be useful to do so now in this session and ensure all the issues have folks assigned with work items.
what if the api changes?
<mzanetti> ideally 2 steps. mock the service, test app with qmltestrunner. have autopilot tests walking through the api
<cjohnston> using an external serivce for testing can provide you with false failures
<mzanetti> 2-step testing... 1) mocks + unit tests, 2) integration tests that that it the api still works
<mzanetti> yeah... you don't need too many AP tests then. Only a few that interact with the server.
how will we know what the failure was caused
remote server testsuite shouldn't block CI
hard to get value out of running tests
Rely on user reports of failures, use mock api's for testing
COPIED FROM PREVIOUS SESSION :-)
ubuntu-
Leo wants to refactor
remember to mockup for network issues
2 tests need to be re-enabled and tested (swipe to delete emulator)
test_
test_
missing tests
test to swipe for different hours in the day
ubuntu-
all good :-)
ubuntu-
forces us to use internet resources which can cause failures
https:/
needs smaller tests, a bit of refactoring
ubuntu-
remove patching home
refactor to use new folder
ubuntu-
refactor to only use local sources
land mp's
https:/
alarm tests
UI alarm test in autopilot
lower level tests for push notifications
ubuntu-
some tests better as unit tests, but they all work and features covered
dropping-
Good, no code changes, tests rock solid
music-app-autopilot 2013/11/20 14:48 8 7 1 0 0 87.5%
shuffle implementation and tests have been an issue
https:/
Needs tests for interfacing with Aritsts, Albums, Songs tabs
https:/
https:/
https:/
https:/
Needs tests for each tab to add an item to a playlist or the queue
https:/
Needs tests to create and modify playlists
https:/
Needs tests for swipe to delete of playlist items and queue items
https:/
calendar-
3 mp's need to land / merge.
once landed all features should have tests
https:/
https:/
ubuntu-
status?
what's development look like?
sudokutouchgame
not in public jenkins view
can we add?
toolbar changed and broke the tests
stock-ticker-
not in public jenkins view
can we add?
https:/
https:/
Work Items
Work items:
[nskaggs] weather app, 2 tests need to be re-enabled and tested (swipe to delete nskaggs): DONE
[nskaggs] add bug for weather app, test to swipe for different hours in the day: DONE
[nskaggs] filemanager, refactor to use new folder: DONE
[nik90] clock, refactor to only use local sources: TODO
[nskaggs] msuic, remove shuffle test: DONE
[vthompson] music, look into shuffle test issues and fix: DONE
music app has lots of new tests to be written: DONE
[nskaggs] calendar, compare victor's mp, merge victor's mp, and potentially merge mine as well: DONE
[chris.gagnon] fix sudokutouch; toolbar issues, et la: TODO
[nskaggs] open bug to add sudokutouch and stock-ticker to the daily image testing, https:/
Dependency tree
* Blueprints in grey have been implemented.
