Community Testing Review

Let's review our current testing efforts across automated and manual testing. In addition, let's discuss the teams and what needs we have, areas for improvement, and review our accomplishments from our previous blueprints.

Testcases / writing / needs
-Manual Tests https://launchpad.net/ubuntu-manual-tests
-Autopilot tests (Core Apps) https://wiki.ubuntu.com/Touch/CoreApps/Testing
for more info on core apps and ci: http://summit.ubuntu.com/uds-1308/meeting/21906/community-1308-app-story-continuous-integration/
-Autopilot tests (gtk apps) https://launchpad.net/ubuntu-autopilot-tests
-Autopilot tests (ubiquity) http://bazaar.launchpad.net/~ubuntu-installer/ubiquity/trunk/files/head:/autopilot/
http://bazaar.launchpad.net/~dpniel/ubiquity/autopilot/files/head:/autopilot/ is the current tests running on jenkins
-autopkg tests http://developer.ubuntu.com/packaging/html/auto-pkg-test.html

Testcase results / viewing / needs
QA Dashboard: http://reports.qa.ubuntu.com
Jenkins
-Ubiquity: https://jenkins.qa.ubuntu.com/view/Saucy/view/AutoPkgTest/job/saucy-adt-ubiquity/
-Autopilot GTK: https://jenkins.qa.ubuntu.com/job/autopilot-ubuntu-applications/
-Autopilot Core Apps: http://reports.qa.ubuntu.com/smokeng/saucy/touch/
-Autopkg: https://jenkins.qa.ubuntu.com/view/Saucy/view/AutoPkgTest/
daniel.holba.ch/autopkgtests/

Teams
laptop-testing
tracker: laptop.qa.ubuntu.com
https://wiki.ubuntu.com/QATeam/Hardware

flavor-qa
http://summit.ubuntu.com/uds-1308/meeting/21839/community-1308-quality-flavors/

Last cycle blueprints:
https://blueprints.launchpad.net/ubuntu/+spec/community-1303-quality-growth
https://blueprints.launchpad.net/ubuntu/+spec/community-1303-quality-rolling
https://blueprints.launchpad.net/ubuntu/+spec/community-1303-quality-testingstrategy
https://blueprints.launchpad.net/ubuntu/+spec/community-1303-quality-ubuntuphone

Autopilot GTK review:
Totem: simple testcase, could use expansion. Format playback support, Playlist support, UI/menu buttons
Nautilus: toolbar is not visible in autopilot; limiting bug. UI/menu buttons. Bookmarking, samba/server connection
Fileroller: contains hack to add files; needs fixed
Firefox: uses process manager
Chromium: help enable upstream tests
Thunderbird: what testing is occuring?
Deja-dup: still needs case
Transmission: still needs case
Network manager: probably not an autopilot test
Empathy: online accounts issue; needs more work
Expanding into non-default applications
pidgin
gnumeric
abiword

Ubiquity review:
Testcases covered:

Running fine, ubuntu one sign on page bug (should be fixed soon)

needs:
dual boot test / crazier partitioning
reinstall, saving /home / upgrades
non-english install
 make sure we aren't grabbing english labels; use objects as much as possible
non-network install / wifi / ethernet

core apps:
https://wiki.ubuntu.com/Touch/CoreApps/Testing
Apps to look at for adding tests
calendar
rss reader
doc viewer
music
All apps need converted to ubuntu sdk
All apps need gate reviews turned on (needs to make sure testcases are robust enough to do so)

Work items:
Continue working through TODO bugs for adding manual tests for flavors (ubuntu gnome, lubuntu, etc): TODO
[nskaggs] finish conversion for all core apps to ubuntu sdk: DONE
[nskaggs] turn on gate reviews for all core apps: DONE
[nskaggs] enable smoke testing for all default core apps (as capacity allows): DONE
[elopio] Make a common base test case for all apps, and propose it to the ubuntu-ui-toolkit: TODO
[dpniel] Open bugs for issues and new todo items found in autopilot ubiquity review: TODO
[dpniel] Open bugs for issues and new todo items found in autopilot gtk review: INPROGRESS
[dpniel] re-write ubiquity test to support non-specific flavor install: INPROGRESS
[dpniel] talk with xnox about U1 dummy account for ubiquity: DONE