tempest

Enable Tempest to generate a clear and informative report

Registered by Yair Fried

A Tempest test run should generate a report that is readable by people outside the Tempest developer group. Currently the report includes the test names with pass/fail, and stack traces for failing tests. this report doesn't say much to people not familiar with the specific tempest tests, let alone non-deveopers.

Proposal:
Since the test is actually a series of assertions, we could use them to generate a tree of module/class/test_case/method with assertions as leaves. Each node will have a pass/fail value, and a Fail node fails all of its ancestors.
This could be further improved by adding documentation to each assertion detailing its purpose as part of the test.
This way, anyone reading this report can see what test is run and what checks it does to determine success.

Blueprint information

Status:
Complete
Approver:
None
Priority:
Not
Drafter:
None
Direction:
Needs approval
Assignee:
None
Definition:
Superseded
Series goal:
None
Implementation:
Unknown
Milestone target:
None
Completed by
Matthew Treinish

Related branches

Sprints

Whiteboard

Example

TestNetworkBasicOps [Pass]
--test_basic
  --...
  --...
--test_re-associate_floating_ip [Fail]
  --detach Floating IP [Pass]
  --re-associate Floating IP [Fail]
    --create_server [Pass]
    --update Floating IP [pass]
    --check_connectivity [Fail]
      --call_until_true(ping ip)[Fail]
         --Fail Details
--test_update_dns [pass]
  ...

Currently this test will return Fail and a stack trace with the AssertionError and text "Timeout ping <IP>"
The proposed report will reveal to anyone unfamiliar with the test code that, while able to test basic connectivity and detaching FLIP from VM, Tempest tried to associate FLIP with a newly created VM but was unable to ping it.

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.