Quality, Testability of Desktop Components

Registered by Jason Warner on 2012-04-25

Goal: improve the level of testing for the desktop

The Canonical upstream teams did some good progresses on testing and quality this cycle, that's a good step for the Ubuntu Desktop quality, we still rely on quite some components from other upstreams though that didn't engage into a such process yet though (those who looked at gnome-settings-daemon, nautilus, gvfs, etc bugs on launchpad probably know what I mean there). I would like to see if we can work on our side and with upstream to get those automated tested in some ways.

It would be also nice to see regular run and report of the testsuits for other components which already have one (i.e glib, gtk) and some testing of their rdepends before upload.

Blueprint information

Status:
Started
Approver:
Didier Roche
Priority:
High
Drafter:
Sebastien Bacher
Direction:
Approved
Assignee:
Sebastien Bacher
Definition:
Approved
Series goal:
Accepted for quantal
Implementation:
Beta Available
Milestone target:
milestone icon ubuntu-12.10
Started by
Sebastien Bacher on 2012-07-11

Related branches

Sprints

Whiteboard

See https://blueprints.launchpad.net/ubuntu/+spec/foundations-p-upload-intermediary for work last cycle on testing rdepends. - ev

Session notes:

 there are autotest plans:
 - run all the tests for package and its rdepends

 idea:
 - upload new gtk, pyogbject, etc to -proposed
 - run the tests
 - move to the archive if tests are ok

 that's integration testing, the vm runs a full session
-> at the end of the build, 1 result file to parse manually and 1 computer parsable file with the result
package for which we want to test the rdepends:
glib
gtk
pygobject
we should have a least basic tests:
- try building a small program
- try running a small C program
- try running a python3-gi gtk program
autopackage tests are not reliable enough to be integrated easily (i.e some tests randomly fail)...
idea of things we should test as well:
test keyboard layout, set one, check that the xorg one really changed, reboot and check it's still the same layout:
test keyboard layout from ubiquity is kept once you log in
test keyboard events (simulate key presses and look at the result):
test ibus:
test the indicators content (dbus introspection of the menus content):
test the "double battery" issue and status changes

(?)

Work Items

Work items:
[pitti] set up "basic" gtk DEP-8 tests (build, run, python-GI binding run): DONE
[pitti] set up "basic" glib DEP-8 tests (build, run): DONE
[pitti] set up gobject-introspection DEP-8 tests (build, scanner/compiler/doc tools): DONE
[pitti] set up pygobject DEP-8 tests (GI with python 2 and 3): DONE
[pitti] glib: make test suite failures fail the build: DONE
[pitti] fix gtk upstream test suite to succeed: DONE
[pitti] gtk: make test suite failures fail the build: DONE
[cyphermox] get started on n-m testing: DONE
[cyphermox] get started on bluez testing: DONE
[cyphermox] investigate tests for e-d-s: DONE
[didrocks] get some basic unity testing running as well on uploads (unity running, upstream unit tests): DONE
[mterry] investigate making lightdm's tests use dep8: DONE
[jibel] send a notification on "job status change" (i.e when a test starts failing) - Documentation https://wiki.ubuntu.com/QATeam/AutomatedTesting/JenkinsEmailNotifications : DONE
[jibel] send a notification on "job status change" - Update existing jobs - (Requires Jenkins 1.447) : POSTPONED
[bryce] investigate xorg testing (fake multi monitor): POSTPONED
[pitti] write archive mounter test for gvfs: DONE
[pitti] discuss gvfs integration test suite with upstream: DONE
[pitti] write smb test for gvfs: DONE
[pitti] write ftp test for gvfs: DONE
[pitti] write sftp test for gvfs: DONE
[pitti] write cdrom test for gvfs: DONE
[pitti] write internal HD partition mounting test for gvfs: DONE
[pitti] write dav test for gvfs: DONE
[pitti] write media player test for gvfs: DONE
[pitti] write USB stick test for gvfs (needs scsi-debug kernel patch which did not land in upstream trunk yet): POSTPONED
[pitti] gvfs DEP-8 tests: DONE
[seb128] investigate testing gnome-settings-daemon: POSTPONED
[jibel] investigate suspend/resume in a VM: DONE
[seb128] talk to the system team about testing indicators integration (sound, message, calendar): TODO
[didrocks] test screensaver and inhibit: DONE

Dependency tree

* Blueprints in grey have been implemented.