Firmware Test Suite Updates for 12.04

Registered by Chris Van Hoof

The Hardware Enablement Team's plans for Firmware Test Suite in the 12.04 cycle.

Blueprint information

Hugh Blemings
Canonical Hardware Enablement
Canonical Hardware Enablement
Series goal:
Accepted for precise
Milestone target:
milestone icon ubuntu-12.04
Started by
Chris Van Hoof
Completed by
Chris Van Hoof

Related branches



= Initial Ideas =

== FWTS ==
* Upstream ownership? Managing new patches + testing.
* good way to collect BIOS related bugs in LP as a reference to implement a test in fwts? public tag like hwe-firmware?
* Continued work on the fwts results parser
* PCIe ASPM BIOS checks
     - see
* Focus on integration with apport (not implemented during Oneiric cycle)
* The Intel PCC interface ( could do with some sanity checking (LP#863175)
       Looked at this, evaluation of PCC Methods generally do SMIs which are impossible to evaluate from userspace. See notes in LP#863175
* The method test could give references to ACPI specifcation when it detects non-compliant AML (LP#859568)
* The method test currently covers most methods/objects, but should be expanded to cover ALL methods/objects over time. We should aim for this over this cycle.
* The battery test should sanity check _BTP via /proc/acpi/battery/*/alarm (LP#853875)
     Scanned 3336 kernel bugs, 462 contained _BTP
* Catching IRQ mis-routing issues.
* Automated kernel error message extraction to help keep klog test in sync with kernel.
* Dump/validate EFI_SYSTEM_TABLE (signature 0x5453595320494249)
     - hard to find this header - it's passed over at boot time and the _init data is lost. Not found a good way
     to search for header in memory (where is it kept? inconsistent across firmware?)
* Continue expanding regression checking of fwts with
* Establish a plan for a pilot program for increasing usage with O{E,D}Ms and BIOS vendors with the goal of getting feedback
* Leverage fwts-live as an easy entry point for gathering this data

== FWTS-Live Image ==
* a clear version to identify the kernel and fwts version contained in the image. should rebuild when there is a new Ubuntu kernel release?
* slimming down image size
* can we leverage Ubuntu Core?
* make it bootable on UEFI systems
* re-basing on Oneiric once released
* bringing up Pango experimental builds
* automated (or easier) submission to
* integrating results parser
* automate opening up nautilus/explorer to appropriate fwts directory on usb media when mounted

== UDS Notes ==
=== Future maintenance ===
* cking transferring to kernel team
* new owner needed for fwts
 * development, packaging, building, etc

 * start a mailing list for fwts development
  * using exsiting list as announcement list
 * new patches to go to new mailing list for review
 * maintainer to apply acked patches
 * continue with normal builds
  * transfer this process to another HWE Team member
 * implement k-t style "2 acks required" process
  * with a new list, keeping ~firmware-testing-team list free for announcements

=== Automated build/test cycle ===
 * cking has been talking to pgraner about automating fwts (fwtsts?)

=== Apport integration ===
 * not a high priority for cking
  * migrate this effort to live image

=== Intel PCC interface ===
 * new test for fwts
  * low priority
  * any systems using this?
  * is it in the kernel? if not, no need to test
 * requires SMIs, so not possible from userspace

=== ACPI method tests ===
 * would like to reference the spec more
  * eg, when finding errors, explain non-compliance
 * currently covers ~80 of exposed methods
  * would like more coverage (about ~30 more)

=== Battery (_BTP) tests ===
 * low priority, not much usage in the wild [ actually, ~13.4% of firmware has _BTP, so still valid ]

=== IRQ misrouting tests ===
 * can we automate this?
  * cking: "I have no idea [how]"

=== procfs ===
 * need to port tests from using /proc to /sys
  * 80% done presently

=== kernel error log parsing ===
 * can vary between kernel versions
 * currently a manual fixup process
 * takes "a morning" per kernel release
 * how much effort to automate? or warn about potential changes?

=== FWTSTS ===
 * git repo online
 * 30% coverage so far
  * for each test case written, another test case to ensure regressions are found is implemented

 * coming soon?

=== Non-x86 build deps ===
 * cking has done a bit of work in this area, and the hope is that we should be able to build for armel now.

== = New Test Requests? ===
  * Nothing new in room :)

 === FWTS Live ===
 * Can we include kernel version and fwts version in iso name
 * Can we continue to update SRU kernels as they're released?
  * Decided to use the latest stable userspace (Oneiric), latest kernel in Development (Precise), and latest fwts version available in fwts-devel PPA
 * Image size: see if others have ideas on how we can further slim down image size (action)
 * Always focus on current stable release (with SRU kernel updates)
  * Using latest Precise kernel builds as they're released, copied to fwts-mainline PPA and rebuild against Oneiric
  * This fixes up a few UEFI related bugs
 * Put together dev build when nearing new release
 * Does this boot on UEFI based platforms with Legacy CSM disabled?
  * It does now :) (vanhoof)

=== FWTS Live Data submission ===
 * Can we leverage Ubuntu Friendly to gather fwts results.log for submission? (checkbox can attach arbitrary files (e.g. fwts logs) to a submission that goes to launchpad. A job description file would have to be written and added/pushed to the checkbox package. Other considerations: privacy implications, depending on how much personal data the logs contain, as I believe all hwdb submissions are publically available. Don't hesitate to ask the checkbox/ certification team, they're happy to help with this).
 * Could we write a apport/checkbox symptoms file to submit results.log rather than an apport hook?
  * If so, could we modify fwts_wrapper to ask the user if they would like to submit the data to Launchpad

=== FWTS Results Parser ===
* Give ayan a ping to see where things are at

=== FWTS Misc ===
* Look into automating how nautilus / win explorer can be opened up automagically when mounted
* Look into file formatting issues Win vs Unix (unix2dos)

Work items for ubuntu-12.04:
[colin-king] Check UEFI CSM: DONE
[colin-king] SMBIOS test should fully sanity check the header and preferably dump the header in information fields: DONE
[colin-king] FACS alignment (LP#869575): DONE
[colin-king] Sort out dependencies for non-x86 builds (LP#861927): DONE
[colin-king] Better DMI decoding scanning. (LP#874373): DONE
[colin-king] Handle deprecated /proc and new (replacement) /sys interfaces correctly for Lucid..Precise support: DONE
[jk-ozlabs] setup patchwork for fwts: DONE
[vanhoof] register mail list for patch traffic, subscribe <email address hidden>: DONE
[colin-king] scan though known tables for _BTP usage: DONE
[canonical-hwe-team] assess time required to automate kernel log message changes?: POSTPONED
[vanhoof] chat with bjf on how fwts-live build is composed for a similar project: DONE
[vanhoof] send out request for folks to see what low hanging fruit exists to slim down fwts-live image size: DONE
[manjo] give build a try on UEFI machines: DONE
[vanhoof] sort out booting fwts-live on UEFI with Legacy CSM disabled: DONE
[canonical-hwe-team] Investigate symptoms file based submission of results.log to launchpad (if feasible, update fwts_wrapper to ask first): POSTPONED
[ayan] check in on nautilus automagically opening right path on usb drive on mount: POSTPONED
[ayan] check into file formatting issues: POSTPONED
[colin-king] to put acpi core engine download / apply script to fwts git repo: DONE
[vanhoof] formally kick off OEM/ODM Pilot with fwts-live: DONE
[alexhung] Detect aspm in FADT table - show warning if it is set: DONE
[alexhung] Detect L0s/L1 aspm errors in PCI configuration space - show warning and/or errors: DONE
[vanhoof] always use the latest kernel from the Ubuntu release in development (Precise): DONE
[vanhoof] investigate how to include the kernel version in the name of the .img (will be supported with live-build v3: DONE


Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.