Managing Manual Testing

Registered by David Murphy on 2009-05-07

Linked to Test Distribution And Scheduling, we need to address requirements for manual tests during automated test sequences.

Blueprint information

Status:
Complete
Approver:
Henrik Nilsen Omma
Priority:
Undefined
Drafter:
Ronald McCollam
Direction:
Needs approval
Assignee:
None
Definition:
Superseded
Series goal:
None
Implementation:
Not started
Milestone target:
None
Completed by
Henrik Nilsen Omma on 2009-06-05

Related branches

Sprints

Whiteboard

============

Machine locking will be covered by the test scheduling spec and test case management by the results tracker spec.

============

Manual test
 * Need to identify low-hanging fruit that we need to add to Checkbox
  * fingerprint scanner
  * video out
  * ethernet hotpluging
  * external devices (usb/firewire, SD cards, etc.)
 * As a policy tests should contain more detail than they do (and should link to the testcase on the wiki?)

Getting new tests
 * for OEM team hardware we can rely on them feeding new tests into Checkbox
 * need to drive community effort for tests on non-OEM hardware

==== Management of testcases ====
Ordering
 * Manual tests should all be lumped together at the beginning of a test run
  * allows user to focus on manual tests and then allow the automatic tests to run and move on to something else

Selection of testcases

/!\ needs to link to User Experience blueprint

 * Should we have a dialog that asks the user what to do?
 * Generate a config file from a website?
 * users should be prompted for what tests to run IFF they have any addons installed... if there are no addons installed just run the default set of tests
 * checkbox-extended package - not installed by default
 * checkbox-addon package - audio tests etc.

Need to have a search feature somewhere -- we need to be able to tell whether a case is in Checkbox already or not

Should have a target number for cases in a suite.
 * currently OEM has ~4-5 per suite
 * need guidelines to keep suites manageable

Need a set of groups of suites
 * Networking, video, etc.

Would be great to show which tests are relevant for the hardware you're running on
 * suite / group selection should be shown after information gathering phase
 * then we can show number of tests in each suite that can be run for this hardware
 * reports should be able to show tests that were skipped due to lacking appropriate hardware
  * do we show this by default?

Checkbox should remember the suites I selected last time
 * would be great to also be able to import/export this list
  * not likely to happen... it will persist but no import/export

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.