Managing Manual Testing

Registered by David Murphy

Linked to Test Distribution And Scheduling, we need to address requirements for manual tests during automated test sequences.

Blueprint information

Henrik Nilsen Omma
Ronald McCollam
Needs approval
Series goal:
Not started
Milestone target:
Completed by
Henrik Nilsen Omma

Related branches




Machine locking will be covered by the test scheduling spec and test case management by the results tracker spec.


Manual test
 * Need to identify low-hanging fruit that we need to add to Checkbox
  * fingerprint scanner
  * video out
  * ethernet hotpluging
  * external devices (usb/firewire, SD cards, etc.)
 * As a policy tests should contain more detail than they do (and should link to the testcase on the wiki?)

Getting new tests
 * for OEM team hardware we can rely on them feeding new tests into Checkbox
 * need to drive community effort for tests on non-OEM hardware

==== Management of testcases ====
 * Manual tests should all be lumped together at the beginning of a test run
  * allows user to focus on manual tests and then allow the automatic tests to run and move on to something else

Selection of testcases

/!\ needs to link to User Experience blueprint

 * Should we have a dialog that asks the user what to do?
 * Generate a config file from a website?
 * users should be prompted for what tests to run IFF they have any addons installed... if there are no addons installed just run the default set of tests
 * checkbox-extended package - not installed by default
 * checkbox-addon package - audio tests etc.

Need to have a search feature somewhere -- we need to be able to tell whether a case is in Checkbox already or not

Should have a target number for cases in a suite.
 * currently OEM has ~4-5 per suite
 * need guidelines to keep suites manageable

Need a set of groups of suites
 * Networking, video, etc.

Would be great to show which tests are relevant for the hardware you're running on
 * suite / group selection should be shown after information gathering phase
 * then we can show number of tests in each suite that can be run for this hardware
 * reports should be able to show tests that were skipped due to lacking appropriate hardware
  * do we show this by default?

Checkbox should remember the suites I selected last time
 * would be great to also be able to import/export this list
  * not likely to happen... it will persist but no import/export


Work Items

This blueprint contains Public information 
Everyone can see this information.