How could we automate certification testing

Registered by Victor Tuson Palau

The current certification testing is largely manual, what approach or method we could follow to make it all or 90% automated?

A possible approach could be to base 90% of certification testing into component testing (both sw and hardware) verified via api drivers (i.e. not UI in applications) and 10% in system use cases. The aim would be for the 90%of the test suite provide a solid confidence level that the system performs well for a given hw (to be run frequently), and 10% to test the integration of component and that the overall user experience remains at the desire level.

And example would be - automated component tests for:
* establish and verifying a network connection
* http/ip stack testing
* data streams
* graphics rendering (displaying an specific pattern that can be capture and easily programmatically validate)
* audio playing (looking for exceptions or panics while performing this task)
* non reporting exception/panics of launching firefox
* file system

manual test case:
* launch firefox and play youtube

Blueprint information

Status:
Not started
Approver:
None
Priority:
Undefined
Drafter:
Victor Tuson Palau
Direction:
Needs approval
Assignee:
None
Definition:
New
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

----------------
* Talk to Kevin (krafty) and determine where the manual test requirements came from
* Spec out a set of tests to be automated;
    - Follow-up on list generated created during the COP
    - Review consumer test recommendations
    - Determine which tests are most expensive to run manually
Examples are:
    - Application tests
    - Network
    - Sound and video

Action Items:
- Move application testing to QA (Ameet to follow-up with Carlos and Marjo)
- For the remaining tests (Network/Sound/Video), complete a high-level review of the tests and address its value (i.e. cost of execution and value to customers) for automation (PS Team)
    - Get feedback from other teams for implementation ideas in 3 test areas from the generated list

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.