bl mp sched test utility pass fail
Test utility to arrive at PASS/FAIL verdict based on repeated iterations of test execution.
The validation approach for scheduler test suite is suggested to be across 2 phases. In the first phase, the test harness executed the selected test case from the suite (core/extended) and provide an initial verdict which indicates if the test execution happened sucessfully without any setup failure etc.
In the first phase, if an execution was successful an output which provides information about the scheduler activity is generated. Multiple iterations of the same test case is repeated and the test output is generated. in the second phase all the outputs from related iterations are parsed and assessed for the PASS/FAIL pattern. And if the number of pass/fail patterns observed for the various iterations is above the agreed threshold, the test case is declared as passed else failed.
Blueprint information
- Status:
- Complete
- Approver:
- Alexander Sack
- Priority:
- Medium
- Drafter:
- Naresh Kamboju
- Direction:
- Approved
- Assignee:
- Naresh Kamboju
- Definition:
- New
- Series goal:
- None
- Implementation:
-
Implemented
- Milestone target:
-
2013.02
- Started by
- Naresh Kamboju
- Completed by
- Naresh Kamboju
Related branches
Related bugs
Sprints
Whiteboard
[nkambo 30 Jan 2013]
notes here.
[nkambo Jan 30 2013]
Test case description is available in below Doc:
https:/
Test results spreadsheet available from the below location:
https:/
big.LITTLE MP scheduler test results have been published in dev-linaro list.
[nkambo Jan 30 2013]
test driver implementation is completed.
Need to integrate and test the test driver with multiple output patterns.
[basil-eljuse 25 Feb 2013]
Latest version of test harness shared do have the capability to declare the PASS/FAIL verdict as reported from individual test cases. The scope of the test harness capability is agreed to be reduced to look only at individual executions rather than having to run multiple times and assess final PASS-FAIL verdict based on multiple runs. With this renewed scope the latest version of the test harness acheives the intended aim.
Updated the blueprint based on request from nkambo.
[nkambo 13 March 2013]
The latest scheduler test suite has PASS/FAIL Verdict capability.
This results have been shared on linaro-dev, power management group and big.LITTLE release team. results spreadsheet link : https:/
Meta:
Headline: Test utility to arrive at PASS/FAIL verdict based on repeated iterations of test execution.
Acceptance:
1. Test harness is able to iterate the execution of a defined test case multiple times, as per an agreed configurable value.
2. Test harness is able to match a defined pattern of observation from the output generated by the test execution and declare the individual instance of the test execution PASS/FAIL
3. Test harness is able to run a second parse where multiple outputs are analysed for pattern matching and declares a PASS/FAIL based on the agreed configurable threshold.
4. This is executed on a TC2 / Android setup
Roadmap id: BIG_LITTLE_MP
Work Items
Work items:
Develop test driver such a way that test driver should take inputs and run number of iterations and given test cases: DONE
Match a defined pattern of observation from the output generated by the test execution and declare the individual instance of the test execution PASS/FAIL: DONE
Run a second parse where multiple outputs are analysed for pattern matching and declares a PASS/FAIL based on the agreed configurable threshold: POSTPONED
Execute test cases on a TC2 platfrom with Android image: DONE
