Managing Manual Testing
Linked to Test Distribution And Scheduling, we need to address requirements for manual tests during automated test sequences.
Blueprint information
- Status:
- Complete
- Approver:
- Henrik Nilsen Omma
- Priority:
- Undefined
- Drafter:
- Ronald McCollam
- Direction:
- Needs approval
- Assignee:
- None
- Definition:
- Superseded
- Series goal:
- None
- Implementation:
-
Not started
- Milestone target:
- None
- Started by
- Completed by
- Henrik Nilsen Omma
Whiteboard
============
Machine locking will be covered by the test scheduling spec and test case management by the results tracker spec.
============
Manual test
* Need to identify low-hanging fruit that we need to add to Checkbox
* fingerprint scanner
* video out
* ethernet hotpluging
* external devices (usb/firewire, SD cards, etc.)
* As a policy tests should contain more detail than they do (and should link to the testcase on the wiki?)
Getting new tests
* for OEM team hardware we can rely on them feeding new tests into Checkbox
* need to drive community effort for tests on non-OEM hardware
==== Management of testcases ====
Ordering
* Manual tests should all be lumped together at the beginning of a test run
* allows user to focus on manual tests and then allow the automatic tests to run and move on to something else
Selection of testcases
/!\ needs to link to User Experience blueprint
* Should we have a dialog that asks the user what to do?
* Generate a config file from a website?
* users should be prompted for what tests to run IFF they have any addons installed... if there are no addons installed just run the default set of tests
* checkbox-extended package - not installed by default
* checkbox-addon package - audio tests etc.
Need to have a search feature somewhere -- we need to be able to tell whether a case is in Checkbox already or not
Should have a target number for cases in a suite.
* currently OEM has ~4-5 per suite
* need guidelines to keep suites manageable
Need a set of groups of suites
* Networking, video, etc.
Would be great to show which tests are relevant for the hardware you're running on
* suite / group selection should be shown after information gathering phase
* then we can show number of tests in each suite that can be run for this hardware
* reports should be able to show tests that were skipped due to lacking appropriate hardware
* do we show this by default?
Checkbox should remember the suites I selected last time
* would be great to also be able to import/export this list
* not likely to happen... it will persist but no import/export
