Certification coverage for 13.04 (and 12.04.3)

Registered by Ara Pulido on 2012-09-03

The Canonical hardware certification team will be hosting a session at UDS to plan and gather feedback about the certification programme for Ubuntu and how coverage will evolve during this cycle.

As both hardware support in Ubuntu and available hardware continue to evolve, we need to review the elements we test, and the amount to which we test them, to ensure certification testing continues to be relevant for modern hardware while de-emphasizing elements whose usefulness decreases with age.

Ubuntu end-users benefit from this effort by having information about which components are tested in Ubuntu-certified systems, which lets them know about the kind of experience they will have out-of-the-box with a Ubuntu-certified system.

OEMs can also benefit from this to ensure that tested components work well with Ubuntu, thus ensuring that their systems can get certified and a good experience is provided for users.

Canonical benefits from a well-defined and discussed certification coverage list to be able to tell OEMs that the hardware they include in their systems is tested to its full ability when certifying for Ubuntu.

== Session overview ==
The UDS session will be a forum where we will discuss:

* A quick overview of a test's lifecycle: blacklist, greylist and whitelist, and how a test moves between them.
* A quick review of existing whitelist tests to determine their relevance and any need for improvement.
* Quick overview of all tests mentioned in the 12.10 coverage document to see how they will change for 13.04, with review of greylist tests to see which ones will make it on the whitelits
* Discussion on new tests to be added (they go in the greylist first)
* Discussion on sleep/resume time thresholds for 13.04

== Current greylist ==

Hibernate/Resume (30 iterations)
MMC data cards
Hybrid Graphics: if UMA or discrete work out of the box: all ports working the certification will include a note specifying which device was certified.
Multiple-Monitor (where supported, multi-head display (2 heads) are tested)
Wi-fi Slider (hardware rfkill)

Secondary function keys:
Media Control
Video switch

Super key (Windows key)
Bluetooth Audio
Vertical edge scrolling
Specific USB 3.0 devices
Lid sensors
Lid open
HDMI audio
Multitouch devices (basic functionality)
Resuming from suspend in less than 3 seconds
Overall suspend/resume time of less than 10 seconds

== Possible new coverage ==

 * Displayport audio
 * Audio settings maintained after suspend. This means ensuring that user-visible settings don't change in ways that will be confusing, like the microphone (or even speakers) muting themselves for no reason, or changes in user-adjustable volume.
 * LEDs
 * Bluetooth file transfer
 * Bluetooth browse file
 * Horizontal scrolling

Blueprint information

Not started
Ara Pulido
Daniel Manrique
Needs approval
Series goal:
Milestone target:

Related branches



== Definition of done ==
* New tests for 13.04 (to go into Greylist, slated for whitelist for 13.10) agreed upon, with scope of coverage and pass/fail criteria.
* Coverage from current greylist that will not be tested anymore for 13.04 agreed upon.
* Coverage from current greylist that will be part of certification (whitelist) for 13.04 agreed upon.
* Coverage from current whitelist that will not be part of certification (i.e. moved to greylist) agreed upon.
* Coverage guide for 13.04 (both desktop and server) published.
* New agreed-upon tests written, tested and integrated into checkbox whitelists.


Work Items

Work items:
"As Ubuntu user, OEM or Canonical enablement, I want to know which devices are tested for 13.04 so I know what I can expect to work when I go shopping for an Ubuntu-certified system / when I want to request certification for a system / when I get a new system to do enablenment on.": TODO
"As hardware certification team, I want to know which devices are interesting and which are no longer too important to OEMs and Canonical enablement, so I can adjust my tests to focus on the most important ones.": TODO
"As checkbox user, I want tests for new coverage items for 13.04 to be present in checkbox, so I can test those items and ensure they work well": TODO
"As hardware certification team, I want to update the list of certification-critical tests (whitelist) with new ones, so I can be sure that my coverage document includes all the goodies I'm testing now": TODO
"As CE QA I want to update my test plans to use the new tests coming from certification, so I know what I'm enabling is what will make systems pass certification": TODO