Service level testing

Registered by Petr Michalec on 2016-08-19

As openstack-salt is being deployed on various systems (rhel, ubuntu) and platforms (vm, baremetal, docker containers) there is a need to properly test service formulas (or openstack component formulas) test individually on these platforms. This blueprint covers approach for standardized developer local test environment as well as it aims to be a part of gerrit/jenkins review process.

This blueprint covers individual formula test scenarios, test & dev tools and approaches for formula validation.

Be aware there is complement blueprint being created. Aimed for system level & integration testing of all salt formula in system context.

Blueprint information

Status:
Started
Approver:
Jakub Pavlik
Priority:
Undefined
Drafter:
Petr Michalec
Direction:
Needs approval
Assignee:
Petr Michalec
Definition:
New
Series goal:
None
Implementation:
Good progress
Milestone target:
None
Started by
Aleš Komárek

Related branches

Sprints

Whiteboard

There were several approaches to be considered, however there are few important requirements to satisfy:

  * formula unit validation
  * python/jinja/yaml syntax validation
  * salt best practices, style validation
  * formula quick smoke tests for integrity
  * standardized dev/test environment to be able to reproduce bugs or work on features
  * formula integration test on various operating systems
  * formula integration test on various operating platforms

Log your statements and proposals at:
https://etherpad.openstack.org/p/salt-service-formula-integration-testing

More to read:
https://app.classeur.io/#!/files/QWTTMuUOYluiAKBBVRAu

# Implementation

## Unit tests

## Syntax, Best practices

Syntax and best practices formula vaidation (ie: for Chef there is an http://www.foodcritic.io/, or Ruby has http://rubocop.readthedocs.io/en/latest/.

Part of the best practices/style validation is a gerrit review process.

## Development and integration testing

(A) - Unit tests
(B) - Quick, smoke tests

There is an Makefile in formula repo with `make test` target. That executes `tests/run_tests.sh` that is capable of quick local smoke testing of formula. This is currently implemented implemented behind gerrit review process.

(C) - Integration tests

More on this topic at: https://app.classeur.io/#!/files/QWTTMuUOYluiAKBBVRAu
(will open for writing if interested)

Gerrit topic: https://review.openstack.org/#q,topic:bp/service-level-testing,n,z

Addressed by: https://review.openstack.org/348237
    Introduce Test Kitchen

Addressed by: https://review.openstack.org/352757
    service, endpoint name to contain region id

Addressed by: https://review.openstack.org/348391
    Introduce test-kitchen

Addressed by: https://review.openstack.org/348338
    Introduce test-kitchen

Addressed by: https://review.openstack.org/347821
    introduce test-kitchen

Addressed by: https://review.openstack.org/347818
    introduce test-kitchen

Addressed by: https://review.openstack.org/347817
    introduce test-kitchen

Addressed by: https://review.openstack.org/347815
    introduce test-kitchen

Addressed by: https://review.openstack.org/347816
    introduce test-kitchen

Addressed by: https://review.openstack.org/373338
    Revert "introduce test-kitchen"

Addressed by: https://review.openstack.org/373339
    Revert "introduce test-kitchen"

Addressed by: https://review.openstack.org/373340
    Revert "introduce test-kitchen"

Addressed by: https://review.openstack.org/373341
    Revert "introduce test-kitchen"

Addressed by: https://review.openstack.org/373342
    Revert "introduce test-kitchen"

Addressed by: https://review.openstack.org/373343
    Revert "Introduce test-kitchen"

(?)

Work Items

Work items:
kitchen yml test suite specification per openstack salt formula: DONE
basic verification scripts per openstack salt formula: INPROGRESS
jenkins and gerrit integration to exec kitchen tests: TODO

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.