Diablo Testing

Registered by Nachi Ueno on 2011-04-07

There are about 600 unit tests and 60 smoke tests. We would like to improve
this for diablo.

- Multiple network configuration which is not only flags but also network structures.
- CI for smoke tests with multiple cluster configuration
- Improve quantities of tests
  1000 unit tests
  150 smoke tests.
- Improve coverage
  Is it possible to check Decision coverage on python?

- Change policy of merging
  We would like to add new rules for merging.
  A) Before proposing merge, the developer must pass smoke tests.
  B) The developer must write 100% coverage unit test, and smoke tests for all use case of new function.

Related Blueprint
Document and reorganize smoketesting code
  https://blueprints.launchpad.net/nova/+spec/testing-smoketests-documentation
 - Move the code in /smoketests to /nova/tests/smoketests/
 - Document the overall structure of the smoketests
 - Document the strategy the smoketests use to setup an environment and perform workflow tests on it
 - Look at creating multiple test fixtures instead of the current usage of flags
 - Link automated smoketest running into Hudson/Tarmac
 - Determine the different configurations (network model, auth model, etc) to use in running smoketests

Improve unittest coverage
  https://blueprints.launchpad.net/nova/+spec/unittest-coverage

Blueprint information

Status:
Complete
Approver:
Vish Ishaya
Priority:
Medium
Drafter:
Nachi Ueno
Direction:
Approved
Assignee:
Monty Taylor
Definition:
Superseded
Series goal:
None
Implementation:
Started
Milestone target:
None
Started by
Thierry Carrez on 2011-07-25
Completed by
Thierry Carrez on 2011-11-17

Sprints

Whiteboard

Also to be discussed during this session:
Create a Multi-Server environment for testing branches, benchmarking and performance metrics for any or all OpenStack projects.

Agreement in Design Summit2011

Add merge rule.
 1. The new code must not lower the coverage rate
 2. The new code must be tested by smoke test with reference archtecture
 3. 95% statement coverage

Vish: Assigning to Monty to implement the new coverage rules

Monty: Pushing this back - will follow up with a detailed implementation plan.

(?)

Work Items