Add unit tests to tempest

Registered by Matthew Treinish on 2013-12-05

We need to verify that internal mechanisms in tempest work and that we do not introduce regressions.

Summit session etherpad:

https://etherpad.openstack.org/p/icehouse-summit-qa-unit-testing

Blueprint information

Status:
Complete
Approver:
Sean Dague
Priority:
High
Drafter:
Matthew Treinish
Direction:
Approved
Assignee:
Matthew Treinish
Definition:
Approved
Series goal:
Accepted for icehouse
Implementation:
Implemented
Milestone target:
milestone icon icehouse-rc1
Started by
Matthew Treinish on 2013-12-05
Completed by
Matthew Treinish on 2014-04-19

Related branches

Sprints

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/unit-tests,n,z

Addressed by: https://review.openstack.org/60378
    Add mock to test-requirements.txt

Addressed by: https://review.openstack.org/60379
    Add unit tests for rest_client

Addressed by: https://review.openstack.org/60578
    Update tempest hacking regarding unit tests.

Addressed by: https://review.openstack.org/60611
    Add test-requirements to all tox jobs

Addressed by: https://review.openstack.org/60619
    Set pipefail for wrapper scripts

Gerrit topic: https://review.openstack.org/#q,topic:bp/config-cleanup,n,z

Gerrit topic: https://review.openstack.org/#q,topic:bug/1260710,n,z

Addressed by: https://review.openstack.org/65006
    Make run_tests.sh for running unit tests

Addressed by: https://review.openstack.org/65007
    Add a run_tempest.sh script

Addressed by: https://review.openstack.org/66945
    Add unit tests for SSH client functionality

Addressed by: https://review.openstack.org/68140
    Unit tests for auth methods in rest client

Addressed by: https://review.openstack.org/67228
    Sync Patch and PatchObject fixtures from oslo-incubator

Addressed by: https://review.openstack.org/68195
    Add unit tests for image waiter

Addressed by: https://review.openstack.org/69341
    Remove unused method basic_auth from rest client

Addressed by: https://review.openstack.org/71890
    Test tempest decorators used on integration tests

Addressed by: https://review.openstack.org/72750
    Ensure that bug number is actually a number for skip_because

Addressed by: https://review.openstack.org/74699
    Couple of fixes to tempest/auth

Addressed by: https://review.openstack.org/74387
    Defines a Credentials class

Addressed by: https://review.openstack.org/76475
    Fixed bug with _error_checker in rest client

Addressed by: https://review.openstack.org/76530
    Fix api_version filter for KeystoneV3AuthProvider

Addressed by: https://review.openstack.org/76531
    Improve tempest auth tests.

Addressed by: https://review.openstack.org/77268
    Split out config option registration

Addressed by: https://review.openstack.org/77269
    Convert fake_config class to use oslo config

Addressed by: https://review.openstack.org/79257
    Add unit test for data_utils

Addressed by: https://review.openstack.org/79572
    Add unit tests for commands

Addressed by: https://review.openstack.org/80416
    Add unit basic unit tests for tenant_isolation

Addressed by: https://review.openstack.org/80647
    Add network credential unit testing

Addressed by: https://review.openstack.org/80704
    Add unit tests for the tempest.common.utils.file_utils

Addressed by: https://review.openstack.org/80742
    Add utils.misc unit tests

Addressed by: https://review.openstack.org/80800
    Add unit tests for wait_for_resource_deletion

Addressed by: https://review.openstack.org/80876
    Add unit tests for NegativeRestClient class

Addressed by: https://review.openstack.org/80878
    Ignore .coverage* files

Gerrit topic: https://review.openstack.org/#q,topic:cleanup-debug,n,z

Addressed by: https://review.openstack.org/80915
    Add unit tests for debug

Addressed by: https://review.openstack.org/80927
    Add unit tests for cli.output_parser

Addressed by: https://review.openstack.org/81075
    Add unit tests for configurable network resources

Addressed by: https://review.openstack.org/81896
    Add config fixture support to unit tests

Addressed by: https://review.openstack.org/82700
    Add unit tests for cli.output_parser

Gerrit topic: https://review.openstack.org/#q,topic:bp/multi-keystone-api-version-tests,n,z

Addressed by: https://review.openstack.org/83620
    Introduce .coveragerc

Addressed by: https://review.openstack.org/83646
    Add unit tests for exceptions

Addressed by: https://review.openstack.org/84086
    Adds unit test for negative class decorator

Addressed by: https://review.openstack.org/84127
    Add unit tests for all generators

Addressed by: https://review.openstack.org/84371
    Rename basic generator test

Addressed by: https://review.openstack.org/84414
    Add unit tests for all generators

Addressed by: https://review.openstack.org/84453
    Remove python25 workaround from glance_http

Addressed by: https://review.openstack.org/84454
    Unit Tests for glance_http

Addressed by: https://review.openstack.org/86405
    Add unittests for remote client

Addressed by: https://review.openstack.org/92172
    small fixes on remote_client return format

Addressed by: https://review.openstack.org/92449
    Check non json type on glance client json_request method

Addressed by: https://review.openstack.org/102605
    Add tests for wait_for_server_status

Addressed by: https://review.openstack.org/108800
    Revert "Add tests for wait_for_server_status"

Addressed by: https://review.openstack.org/120824
    Change LOG.exception to LOG.info for skipped tests

Addressed by: https://review.openstack.org/120825
    Fix safe_setup decorator to not call tearDownClass for skipped tests

Addressed by: https://review.openstack.org/120826
    Add unittests for safe setup decorator

(?)

Work Items

Work items:
Rest client: INPROGRESS
Code that is triggered on handling 500 errors or timeouts: TODO
Utility scripts: INPROGRESS
Decorators: DONE
SSH/Remote client (jaypipes): INPROGRESS
Stress test framework: INPROGRESS
XML vs. JSON : TODO
Check that XML is generated by XML clients: TODO
Ensure that XML tests actually generate XML: TODO
Add documentation to enable unit testing(mtreinish): INPROGRESS
Isolated Creds(mtreinish): INPROGRESS

This blueprint contains Public information 
Everyone can see this information.