Port checkbox-certification-server on top of plainbox

Registered by Zygmunt Krynicki on 2013-04-24

checkbox-certification-server is a set of plugins, whitelists and
configurations for checkbox that define the tests to be performed for
server certification.

It has a few characteristics that make it a good candidate to start
using the new plainbox testing runner/engine. (Find a link to the
plainbox blueprint for a quick introduction):
- It's usually run only in very controlled environments
- Usually run by a trained technician who can be relied on to report and
  help diagnose any problems.
- Most of the tests are automated, a few manual ones ask simple
  questions that don't require complex interactions.
- At the end, it should submit the results to the certification website
  and output a local XML report. (Do we want to include outputting HTML
  here?)

Some components are missing for plainbox to fill this role. A user
interface and a secure way of running root-level jobs are examples.

This blueprint should result in a package that can be installed and a binary that can be run, to execute the server certification test suite, with a simple UI to ask the user for test verifications, and that can at the end submit results to the certification web site as well as output a locally-openable report, in XML or preferrably HTML.

Tackling the work needed to run these tests would carry the following
benefits:

- Creation of a prototype using the plainbox libraries, toolset and
  architecture to build an actual, user-interactive application results
  in a better understanding of Plainbox architecture, so developers get
  the experience needed to tackle more difficult developments (such as
  an eventual end-user-oriented desktop testing app based on plainbox)

- Use of a better-documented, more robust and better tested engine for
  certification tests.

- End user (i.e. testers) benefit from having a more robust, simpler
  tool. It's also faster and easier to adapt to their requests and
  needs.

- Developers benefit from having plainbox as the testing core, since
  behavior changes or improvements are easier to test and implement,
  and plainbox makes test development and debugging easier.

- The work which will be potentially needed to organize and repackage
  tests, even if it's only discussed during the 3-month cycle, will
  enable better separation of checkbox tests and dependencies, which is
  already being requested by some of our other efforts (cloud testing,
  checkbox in Ubuntu server)

Topics to consider:
    - urwid as possible ui?
      Checkbox already abstracted the UIs, providing a simple 'API' for
      UI classes to implement, this made adding new UIs potentially
      simple. However, care needs to be taken with the data structures
      that are sent back and forth (see a few bugs on the QT ui that
      stem from assuming data is in one format (e.g. dictionary) and
      then receiving something else (some missing keys, or a list)).

      Also, this abstraction limited the possibilities of the UI, which
      is one of checkbox's shortcomings that plainbox aims to remedy.

      So while some care should be taken to make the UI code as
      pluggable as possible, I think part of the emphasis should be
      instad in making the plainbox libraries easy to use from whichever
      UI design we choose to implement.

Blueprint information

Status:
Complete
Approver:
None
Priority:
Undefined
Drafter:
Zygmunt Krynicki
Direction:
Needs approval
Assignee:
None
Definition:
Approved
Series goal:
Proposed for trunk
Implementation:
Implemented
Milestone target:
milestone icon 2013-05-23
Started by
Zygmunt Krynicki on 2014-05-01
Completed by
Zygmunt Krynicki on 2014-05-01

Related branches

Sprints

Whiteboard

ZK:
We need to look at how to move c-c-s on top of plainbox.
With the recent development on plainbox sru, we seem to have most of the backend logic ready. The challenge will be in putting that together to a more interactive user interface.

We also need to decide how to ensure continuity, should we have checkbox-certification-server-ng co-existing with the current program (that would be a safer alternative to a flag day). Since checkbox (as a command name) is unused, we could perhaps use it for all future plainbox-based checkbox programs.

SP:
The link introducing plainbox: https://blueprints.launchpad.net/ubuntu/+spec/client-1303-new-checkbox-core-plainbox

(?)

Work Items

Work items:
As a PlainBox developer I want to release PlainBox 0.3 so I can baseline all the code that was needed to perform SRU - M: DONE
As Marc Deslauriers I want PlainBox to execute jobs requiring root privileges in a secure way so I can allow ubuntu CheckBox to use PlainBox as its new core - L: TODO
As a PlainBox developer PlainBox I want to implement the secure way of running jobs as root (as agreed with the Security team) so I can run my tests without (SRU) or with just one password prompt (self-cert) - L: DONE
As a SRU Test Coordinator I want PlainBox to have the logging feature so I can debug remote systems - L: TODO
As a PlainBox developer I want to create a "CheckBox" application so I can provide an executable that has stable command line interface and production features vs PlainBox as the development tool - L: INPROGRESS
As a PlainBox developer I want to study how it's possible to reuse existing checkbox ui code so I can decide if a story to actually implement it is doable - M: TODO
As a PlainBox developer I want to reuse existing checkbox ui so I can minimize the work to be done to get certification packages using PlainBox core - L: TODO
As Ara I want a release of checkbox-certification-server using plainbox core so I can see the benefit of all the team work on plainbox - XL: TODO

This blueprint contains Public information 
Everyone can see this information.