Managing Manual Testing

Registered by David Murphy on 2009-05-07

Linked to Test Distribution And Scheduling, we need to address requirements for manual tests during automated test sequences.

Blueprint information

Henrik Nilsen Omma
Ronald McCollam
Needs approval
Series goal:
Not started
Milestone target:
Completed by
Henrik Nilsen Omma on 2009-06-05

Related branches




Machine locking will be covered by the test scheduling spec and test case management by the results tracker spec.


Manual test
 * Need to identify low-hanging fruit that we need to add to Checkbox
  * fingerprint scanner
  * video out
  * ethernet hotpluging
  * external devices (usb/firewire, SD cards, etc.)
 * As a policy tests should contain more detail than they do (and should link to the testcase on the wiki?)

Getting new tests
 * for OEM team hardware we can rely on them feeding new tests into Checkbox
 * need to drive community effort for tests on non-OEM hardware

==== Management of testcases ====
 * Manual tests should all be lumped together at the beginning of a test run
  * allows user to focus on manual tests and then allow the automatic tests to run and move on to something else

Selection of testcases

/!\ needs to link to User Experience blueprint

 * Should we have a dialog that asks the user what to do?
 * Generate a config file from a website?
 * users should be prompted for what tests to run IFF they have any addons installed... if there are no addons installed just run the default set of tests
 * checkbox-extended package - not installed by default
 * checkbox-addon package - audio tests etc.

Need to have a search feature somewhere -- we need to be able to tell whether a case is in Checkbox already or not

Should have a target number for cases in a suite.
 * currently OEM has ~4-5 per suite
 * need guidelines to keep suites manageable

Need a set of groups of suites
 * Networking, video, etc.

Would be great to show which tests are relevant for the hardware you're running on
 * suite / group selection should be shown after information gathering phase
 * then we can show number of tests in each suite that can be run for this hardware
 * reports should be able to show tests that were skipped due to lacking appropriate hardware
  * do we show this by default?

Checkbox should remember the suites I selected last time
 * would be great to also be able to import/export this list
  * not likely to happen... it will persist but no import/export


Work Items

This blueprint contains Public information 
Everyone can see this information.