How could we automate certification testing

Registered by Victor Tuson Palau on 2010-10-22

The current certification testing is largely manual, what approach or method we could follow to make it all or 90% automated?

A possible approach could be to base 90% of certification testing into component testing (both sw and hardware) verified via api drivers (i.e. not UI in applications) and 10% in system use cases. The aim would be for the 90%of the test suite provide a solid confidence level that the system performs well for a given hw (to be run frequently), and 10% to test the integration of component and that the overall user experience remains at the desire level.

And example would be - automated component tests for:
* establish and verifying a network connection
* http/ip stack testing
* data streams
* graphics rendering (displaying an specific pattern that can be capture and easily programmatically validate)
* audio playing (looking for exceptions or panics while performing this task)
* non reporting exception/panics of launching firefox
* file system

manual test case:
* launch firefox and play youtube

Blueprint information

Status:
Not started
Approver:
None
Priority:
Undefined
Drafter:
Victor Tuson Palau
Direction:
Needs approval
Assignee:
None
Definition:
New
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

----------------
* Talk to Kevin (krafty) and determine where the manual test requirements came from
* Spec out a set of tests to be automated;
    - Follow-up on list generated created during the COP
    - Review consumer test recommendations
    - Determine which tests are most expensive to run manually
Examples are:
    - Application tests
    - Network
    - Sound and video

Action Items:
- Move application testing to QA (Ameet to follow-up with Carlos and Marjo)
- For the remaining tests (Network/Sound/Video), complete a high-level review of the tests and address its value (i.e. cost of execution and value to customers) for automation (PS Team)
    - Get feedback from other teams for implementation ideas in 3 test areas from the generated list

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.