Add Reporting Facility in Rally

Registered by Chandra Ganguly on 2015-01-27

Rally is a great tool to get perf/scale numbers; However when we have a lot of testcases, its hard to read through the entire logs to get a summary of view of what ran and what failed; The proposal here is to put a command line option post rally execution (similar to the json print option) to summarize the testcases ran, and the pass/fail stats; Also on failure the enhancement should list out the failure reasons against each testcase. If this is added, one can even run functionality test from rally and get a summary of the results. This includes 3 work items:

1) Add "rally task show" command that actually prints a summary of the details
2) Make FAIL/SUCCESS based on SLA checks
3) Add special argument to "rally task show" --print-errors. That will print traces

Blueprint information

Status:
Not started
Approver:
Boris Pavlovic
Priority:
Medium
Drafter:
Chandra Ganguly
Direction:
Approved
Assignee:
Chandra Ganguly
Definition:
Approved
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

The idea is to use Rally for system sanity every night and we chain together scenarios in one file. So we need something that pretty prints a sequence of scenarios. An example below shows each scenario and overall status without having to go through all the json output etc. This is intended not for 1 scenario but when rally is used for a chaining together scenarios and using it for PASS/FAIL.

Results Summary
---------------
Total : 4
Passed: 2 [50%]
Failed: 2 [50%]

+-----+------------------------------+----------------+--------+
| # | Scenario Name | RunTime | Result |
+-----+------------------------------+----------------+--------+
| 1.1 | Authenticate.keystone | 0.141638040543 | PASS |
| 1.2 | Authenticate.keystone | 0.135577201843 | PASS |
| 2.1 | Authenticate.validate_cinder | 0.222651004791 | FAIL |
| 2.2 | Authenticate.validate_cinder | 0.173644065857 | FAIL |
+-----+------------------------------+----------------+--------+
Note: **: Throughput in bps is listed instead of RunTime
     Also, Pass/Fail for throughput test is based on 5% tolerance if
     no exception found

 FAILED TestCases
+-----+------------------------------+--------------------------------------------+
| # | Failed Test Name | Failed Logs |
+-----+------------------------------+--------------------------------------------+
| 2.1 | Authenticate.validate_cinder | -+- [u"<type 'exceptions.NameError'>" |
| | | -+- "global name 'LOG' is not defined" |
| | | -+- 'Traceback (most recent call last):\n |
| | | File "/usr/local/lib/python2.7/dist- |
| | | packages/rally/benchmark/runners/base.py", |
| | | line 84, in _run_scenario_once\n |
| | | method_name)(**kwargs) or |
| | | scenario_output\n File |
| | | "/usr/local/lib/python2.7/dist-packages/ra |
| | | lly/benchmark/scenarios/authenticate/authe |
| | | nticate.py", line 75, in validate_cinder\n |
| | | LOG.error(\'erroring out\')\nNameError: |
| | | global name \'LOG\' is not defined\n'] |
| | | |
| 2.2 | Authenticate.validate_cinder | -+- [u"<type 'exceptions.NameError'>" |
| | | -+- "global name 'LOG' is not defined" |
| | | -+- 'Traceback (most recent call last):\n |
| | | File "/usr/local/lib/python2.7/dist- |
| | | packages/rally/benchmark/runners/base.py", |
| | | line 84, in _run_scenario_once\n |
| | | method_name)(**kwargs) or |
| | | scenario_output\n File |
| | | "/usr/local/lib/python2.7/dist-packages/ra |
| | | lly/benchmark/scenarios/authenticate/authe |
| | | nticate.py", line 75, in validate_cinder\n |
| | | LOG.error(\'erroring out\')\nNameError: |
| | | global name \'LOG\' is not defined\n'] |
| | | |
+-----+------------------------------+--------------------------------------------+

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.