New CheckBox Core - PlainBox

Registered by Zygmunt Krynicki

CheckBox is a tool used by the Hardware Certification (HW Cert) team to test and certify laptops, desktops and servers with Ubuntu. CheckBox is also used for some aspects of testing Unity and will most likely be used for the revitalized Ubuntu Friendly initiative.

The last UDS saw discussion on the issues related to the technology behind CheckBox:

https://blueprints.launchpad.net/certify-planning/+spec/cert-r-checkbox-simplification

At that time there was no concrete proposal on how to address those issues and improve quality, speed or maintainability. During the last few months the HW Cert team has made large inroads into replacing the core of checkbox with new, better architecture known as PlainBox.

We would like to use UDS to share our progress with other teams in Canonical as well as community members who may be interested in using CheckBox or may be more interested in PlainBox itself.

Blueprint information

Status:
Not started
Approver:
Ara Pulido
Priority:
Undefined
Drafter:
Zygmunt Krynicki
Direction:
Needs approval
Assignee:
None
Definition:
Discussion
Series goal:
None
Implementation:
Informational Informational
Milestone target:
None

Related branches

Sprints

Whiteboard

CheckBox Technology Stack:

https://docs.google.com/drawings/d/1OcbXKuvjscDa_6k-rDhaqMoQonyHZ6tp8PNjWVNBVWw/edit?usp=sharing

Topics to cover:

* CheckBox at UDS-Q
  - started from a non-trivial, mature project
  - daily usage in several different scenarios
  - downstream users affected by bugs and API changes
  - not testable in current form
  - difficult to test in practice (hardware specific, many manual tests)

* PlainBox at UDS-1303
  - new project developed inside lp:checkbox tree
  - new core with good architecture and lots of features that make it a better checkbox
  - better development process, semantic patches, tests, docs, API design
  - all merging done by a CI loop that does useful tests (jenkins + tarmac)
  - lots of tests in the core, 80% + code coverage
  - few integration tests for checkbox scripts, lots of work needed here
  - initial docs, auto-built from source at http://plainbox.rtfd.org/
  - core is mostly feature complete, runs all checkbox jobs
  - daily development ppa available
  - initial deployment as checkbox replacement for SRU testing, just after this UDS

[achiang] what is a "semantic patch"?
[zkrynicki] a patch that brings a logical change rather than an artifact of using a version control system. in practical terms our history is pretty and has less "fix typo", "fix crap" and more "add feature", "alter function signature", "add documentation" type of changes. When a merge request gets feedback and needs to change it won't get those changes as a series of "fix stuff" patches on top, instead we'll rewrite the patches that were proposed to make following the changes much easier for both reviewers and any other users.

* Next 3 months:
  - extend integration tests to run most of checkbox jobs
  - some more work on the core
  - see what we learn what SRU testing
  - abstract away all of checkbox job specifics to be able to fix job design later

* Large TODOs to start soon:
  - new job/script project with better job quality, tests, docs and new job format
  - new GUI for plainbox (maybe based on new Ubunut QT SDK? Hopefully pure-python)

* Downstream users:
  - what this means for you
  - tell us your story, how are you using checkbox technology today
  - how can you be affected by checkbox evolution

[achiang] as a downstream user of checkbox, one issue we hit all the time is the lack of separation between the checkbox "test runner" and the actual system tests. As a downstream, I would love to have fixes in the "test runner" without disturbing the tests I have selected for my project. The fact that the test runner and the system tests are delivered in packages that move in lockstep with each other make this difficult.

[zkrynicki] This is essentially solved but you cannot get the benefits of that yet. Checkbox will be composed of three sub-projects with separate releases: chceckbox -- the stuff that can be used for certification, probably having a custom GUI, plainbox the core that fuels all of that, and (name-still-secret) that has all of the job definitions and scripts. Downstream users can then pick a stable plainbox release, any (name-still-secret) job definition release package, add more jobs from any source (project specific jobs), and either bundle a custom gui (maybe forked from the gui rewrite planned for TODO in my notes above) or just go with the command line plainbox client.

[cwayne] I cannot +1 achiang's comment more. Furthermore, I was wondering what the future of CheckboxEditor would be? Would something like that still be supported? Will there be a system for creating downstream versions of checkbox for specific applications/projects? Will plainbox be as strict about syntax for jobs as checkbox is? Will plainbox be able to run fully-automated from start to end? Will it be able to be run on a host system and run tests on a remote system?

[ara] We don't have plans to add new features to Checkbox Editor. It would be good to know the features that you guys use from editor and see alternatives to that. Checkbox already can run fully automated, and plainbox will allow that as well. We currently don't have plans to support remote testing, but we would gladly accept patches for that. One of the benefits of this redesign of the core is that it is much easier to understand and well tested, so participation in the project is easier.

[zkrynicki] We don't have a roadmap on checkbox editor. This is something we should talk about during the session or if needed, create a dedicated session (even after UDS) to discuss. There will be full support for creating custom versions of checkbox but to get there we need to fix a few issues that largely prevented us from doing it properly before. We'll clearly separate private unstable API that we can and will break from stuff that we will support for a very long time. Plainbox almost supports additional job providers so apart from the job format and semantic that checkbox has (the only one currently implemented) we will support a new job format that is 99% the same, except for how it handles local jobs (those will be equally powerful but better from architecture point of view). Determined users can also create entirely new job formats with custom logic to a level that was impossible before. I can discuss this in the session briefly but I would encourage a dedicated architecture session / followup engineering sessions to explain that better. Plainbox already supports fully automated runs and is much better at it than checkbox was. As for testing remotely on one machine to another. We don't have anything special planned but one of the GUI ideas we discussed includes running GUI as webapp (that is runinng on DUT) and connecting to it from another machine via a browser. There are many challenges in that design but it could be most powerful for testing new form factors and for cross-platform users). Also, lastly, the certification team already maintains one custom checkbox derivative so we know all the issues associated with that style of projects. We want to make that as painless as possible in the future.

[cprofitt] How does PlainBox affect UbuntuFriendly?

[zkrynicki] PlainBox will allow UbuntuFriendly to build a customized application that reuses checkbox jobs (or bundles additional custom jobs) and provides a simple API to run the desired tests. Resulting data can be exported to a number of supported formats or to any custom format with custom exporter (that is easy to build on top of our current exporter stack). Note that currently plainbox does _not_ have a public API so if such convergence is desired it should be developed in the lp:checkbox tree to ensure API transitions are handled without issues.

(?)

Work Items