Automatically create negative tests out of schemas

Registered by Marc Koderer

The number of negative tests are steady growing in tempest. Most of them are quite simple and just test a single argument with flawed input. This blueprint will use schemas to make it possible to express tests like we have in a declarative manner and to auto-generate negative tests as well. There is another blueprint that addresses moving the current negative tests into separate files: https://blueprints.launchpad.net/tempest/+spec/negative-test-files.

Blueprint information

Status:
Complete
Approver:
David Kranz
Priority:
High
Drafter:
David Kranz
Direction:
Approved
Assignee:
None
Definition:
Approved
Series goal:
Accepted for icehouse
Implementation:
Implemented
Milestone target:
milestone icon icehouse-rc1
Started by
David Kranz
Completed by
David Kranz

Related branches

Sprints

Whiteboard

See https://blueprints.launchpad.net/tempest/+spec/negative-test-files for separate files for negative tests

Gerrit topic: https://review.openstack.org/#q,topic:move/negative-tests,n,z

Addressed by: https://review.openstack.org/64089
    Moves negative tests from tempest/api/compute/limits/test_absolute_limits

Gerrit topic: https://review.openstack.org/#q,topic:bp/negative-tests,n,z

Addressed by: https://review.openstack.org/64733
    Autogeneration framework for negative tests

Addressed by: https://review.openstack.org/65930
    Add unit tests for negative test framework

Addressed by: https://review.openstack.org/71579
    Negative tests: Add result check for resources

Addressed by: https://review.openstack.org/72667
    Adpat documentation for negative testing

Addressed by: https://review.openstack.org/77823
    Add support for negative tests with admin client

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.