Checkbox Certification for ARM Servers

Registered by Brendan Donegan

We want to use Checkbox to certify ARM based servers. Previous investigation has shown that Checkbox requires some fixes to make it work well on ARM. The goal of this blueprint is to define the work that needs to be done to achieve this.

Being a tool for testing hardware, many of the tests used in Checkbox have succumbed to the temptation of using x86 PC specific tools for implementation. These tests need to be identified and a suitable platform agnostic alternative found where possible. Also there may be incompatible data generated in the test submission that prevents processing by the certification website where the test results need to be stored. These problems need to be fixed in the first instance. There is a list of already open bugs which may impact ARM server testing in Checkbox: http://tinyurl.com/arej6q9. Fixes need to be applied for at least the latest LTS (Precise) and the latest development release.

In addition, the certification site (https://certification.canonical.com) itself may need some modifications to store information on the hardware being certified in an appropriate way. The public-facing certification site (http://www.ubuntu.com/certification) may also need changes to display this information in an appropriate way.

By the end of the work we plan to be able to run checkbox-certification-server on an ARM target with a valid submission generated and no false positive test failures.

Blueprint information

Status:
Not started
Approver:
Ara Pulido
Priority:
Undefined
Drafter:
Brendan Donegan
Direction:
Needs approval
Assignee:
None
Definition:
New
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

ZK: things that I know may need changes:
1) Anything we send / don't send as a part of submission.xml
2) Anything that may be borked by different /proc and /sys data (this includes dmi)
3) Anything external that we depend on that only exists for x86 (fwts?)

As for checkbox itself, I _suspect_ it's okay and works just fine on ARM. We may want to create a CI system similar to the one we use with plainbox now but based on ARM devices.

BD:

Note that it may be possible to use plainbox instead assuming we get that to a fit state for running checkbox-certification-server. However most of the work lies in the jobs, so that probably won't help us that much.

ZK:

IIRC Checkbox "works" well enough on ARM so this won't be a problem. We ran it on some ARM boards and it was okay with the exception of the DMI job

BD:

We need to identify the target hardware for this work.
As implied by the blueprint title, we will only be interested in fixing jobs run by server-cert.whitelist in checkbox-certification

(?)

Work Items

Work items:
As a developer of Checkbox, I want to gain access to one or more ARM servers, so that I can find out what is broken on that platform and fix it - M: DONE
As a developer of Checkbox, I want to ensure I know which releases to work on (Precise + latest kernel?) and any specific build to use, so that I can focus my efforts on a specific version of Ubuntu - S: DONE
As a user of Checkbox on an ARM server, I want to run Checkbox with the desired whitelist and check what is broken, raising a bug for each issue identified, so that I know which issues will need to be fixed before I can use it - M: TODO
As the hyperscale team I want to know how Checkbox uses DMIdecode and other x86-specific tools so that we can provide advice on alternative places to get this same information - M: TODO
As a user of Checkbox on an ARM server, I want Checkbox (including the core/plugins and any job commands) to be fixed to generate a valid submission for the certification site, so I know at least one part of the process is working - L: TODO
As a user of Checkbox on an ARM server, I want to coordinate with the infrastructure team on any changes needed to allow the certification site to accept ARM submissions, so that I can make submissions there and they will be displayed correctly - L: TODO
Check if access to the management engine is required to run some server tests (IPMI?): TODO
As the hyperscale team I want there to be a test for MaaS installation (automated if possible) and for the node to be installed with MaaS during testing so that we fully test the solution that we are offering - M: TODO
As a developer of Checkbox, I want to setup a CI loop on daily packages, so that I can see immediately when something is fixed (or broken) - M: TODO
As a user of Checkbox on an ARM server, I want to fix each script/job (or other part of Checkbox) which is broken, so that I can run Checkbox in confidence that the tests will work - XL: TODO