X Testing Infrastructure

Registered by Bryce Harrington on 2007-11-22

This specifies a basic framework to use for doing manual, semi-automated, and automated testing of X.org and related bits.

Several side benefits include: a detailed debugging procedure handbook that will help Canonical support; tests that can be contributed upstream; and automated builds/packaging of newly released video and input drivers that users needing driver updates can easily access.

This is not really intended to be an elaborate all-in-one fully automated test harness, but rather just a collection of tools that enable X testers to work better and faster, and communicate results more easily.

This is a long-term development effort that will probably span several distro releases before it can be considered mature.

Blueprint information

Status:
Complete
Approver:
Martin Pitt
Priority:
High
Drafter:
Bryce Harrington
Direction:
Needs approval
Assignee:
Bryce Harrington
Definition:
Approved
Series goal:
Accepted for karmic
Implementation:
Implemented
Milestone target:
None
Started by
Bryce Harrington on 2007-11-22
Completed by
Bryce Harrington on 2009-11-24

Whiteboard

bryce 2007-11-21: High level description of a planned testing framework for Ubuntu's X.org. It is expected to take a few releases before it's mature, but many bits and pieces already exist, and it's hoped some portion can be running for hardy.

bryce 2007-12-19: There was some feedback left here that unfortunately got deleted when I tried giving feedback to it, so I'll leave it here in case the commenter checks back. The commentary was about having a more robust failsafe fallback situation, rather than about X testing infrastructure, so not directly relevant to this spec. However, the good news is that Debian (perhaps inspired by our failsafe work) is looking into building in a failsafe capability to the xserver itself, which will provide a better all-around solution, since many of X's failure modes are hard to catch automatically.

bryce 2007-12-19: Progress has been made towards the X Auto Packager, which is packaged and available as xorg-pkg-tools. It has been tested for drivers, libs, and protos. I've tried using it for packaging xserver's git-head, however this has proven to be too complex due to dependencies on git-head versions of mesa, missing dependencies, and so on; we'll get the biggest bang for the buck in testing git-head versions of drivers, so my focus be limited to those for the time being. xorg-pkg-tools can be obtained here:

    https://code.edge.launchpad.net/~xorg-edgers/xorg-server/xorg-pkg-tools

2008-11-18 bryce: Draft of XSmoke test available with several basic tests at

    https://code.edge.launchpad.net/~ubuntu-x-swat/xorg-server/xsmoke

2008-12-12 bryce: Discussed at UDS, generally positive. xsmoke received packaging from cr3.

2008-12-15 pitti: approval ack

2009-06-01 pitti: Is there something left to do for Karmic?

2009-11-24 bryce: I think the desktop-lucid-xorg-triaging-diagnosis can be considered the sequel to this for remaining work items. Also, much of the test infrastructure stuff described in this blueprint is already provided by checkbox, so no need to reinvent that wheel. We still need better X tests for checkbox, but I decided rather than take for writing the tests on my overly full plate, I'd just document the testing steps at http://wiki.ubuntu.com/X/Testing/ and leave the scripting to the QA experts.

So anyway, I think we can close out this one, and focus effort on desktop-lucid-xorg-triaging-diagnosis

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.