Provide a common Validation Framework inside python-tripleoclient

Registered by Cédric Jeanneret on 2018-07-18

Track work and changes in order to provide a common, unified validation framework inside tripleoclient. It will help validating environments prior a deploy (both undercloud and overcloud), and prior any update/upgrade of said environments.

Blueprint information

Status:
Not started
Approver:
Emilien Macchi
Priority:
High
Drafter:
Cédric Jeanneret
Direction:
Needs approval
Assignee:
Cédric Jeanneret
Definition:
Approved
Series goal:
Accepted for train
Implementation:
Unknown
Milestone target:
milestone icon train-3

Related branches

Sprints

Whiteboard

[2019-04-15] (aschultz) As this is not completely implemented, moving to Train.

Trello Board: https://trello.com/b/x3h5FrnX/validation-framework

Gerrit topic: https://review.openstack.org/#/q/topic:bp/validation-framework

General work items description:

* Allow executing TripleO validations using pure Ansible from the Undercloud, this is because we should not rely on Mistral to run the validations in the Undercloud.

!! TODO

Main commands:

* Allow running any of the installed validations
   openstack validator run <OPTIONS>

* Show all the installed validations, displaying the description parameter for each one.
   openstack validator list-validations <OPTIONS>

* Run a wide system self-check in order to ensure it has the proper services to run the other validations (ansible is working, playbooks are valid, Mistral is running and in good shape, and so on):
  openstack validator self-check <OPTIONS>

Options

* The framework should allow selecting between Mistral and Ansible for running the validations with an additional set of options:
  openstack validator [--use-ansible] Default: --use-ansible (we're in CLI)

* Provide options for running the validations:
   openstack validator run [--group_validation | --validations | --validation]

* Allow running custom self-maintained validations:
   openstack validator run [--extra-playbooks]

* Provide a method to run all Overcloud or Undercloud validations ***This might not be needed as the validations itself are targeted.
  openstack validator run --overcloud

* Select the output style for any of the CLI commands: --format [json | nagios | table] Default: json

Storage/Logging

* Just like mistral keeps a record of workflow executions, the framework should log results of validation runs in a log file or some other (permanent? ephemeral?) storage system.

Documentation

* Generate a docs section with examples and how the validations are and should be executed from this new framework

!! Later enhancements

* Run a minimal report giving information about the system, this can provide information like the OpenStack version running and i.e. if the deployment is containerized or not, also, the current services status.
  openstack validator get-report <OPTIONS>

* Mistral support: --use-mistral (maybe implemented earlier than thought)

Addressed by: https://review.openstack.org/589169
    Validation Framework specifications

Gerrit topic: https://review.openstack.org/#q,topic:bp/validation-framework,n,z

Addressed by: https://review.openstack.org/636283
    Amend Validation Framework Spec

Gerrit topic: https://review.openstack.org/#q,topic:ansible/without/mistral,n,z

Addressed by: https://review.openstack.org/636581
    Add new CLI option openstack tripleo validate list

Gerrit topic: https://review.openstack.org/#q,topic:bug/1815725,n,z

Addressed by: https://review.openstack.org/636567
    Add missing queue_name input parameter for listing validations

Addressed by: https://review.openstack.org/636930
    Add new CLI option openstack tripleo validate run

Addressed by: https://review.openstack.org/636958
    Migrating Undercloud playbooks to their own Ansible Role

Addressed by: https://review.openstack.org/637485
    Add missing queue_name input parameter for listing validations

Addressed by: https://review.openstack.org/638426
    Add unit tests for running and listing validations from the CLI

Addressed by: https://review.openstack.org/638661
    Add dhcp-validations role

Addressed by: https://review.openstack.org/639093
    Add haproxy role

Addressed by: https://review.openstack.org/639100
    Add ovs-dpdk-pmd

Addressed by: https://review.openstack.org/639103
    Add pacemaker-status role

Addressed by: https://review.openstack.org/640015
    Add dns role

Addressed by: https://review.openstack.org/640036
    Add repos validations role.

Addressed by: https://review.openstack.org/640046
    Add ceilometerdb-size role

Addressed by: https://review.openstack.org/640075
    Add ntpstat role

Addressed by: https://review.openstack.org/640112
    Add openshift-on-openstack role

Addressed by: https://review.openstack.org/640124
    Add rabbitmq-limits role

Addressed by: https://review.openstack.org/640129
    Add nova-event-callback role

Addressed by: https://review.openstack.org/640131
    Add mysql-open-files-limit role

Addressed by: https://review.openstack.org/640135
    Add ceph role

Addressed by: https://review.openstack.org/640160
    Add xfs-check-ftype role

Addressed by: https://review.openstack.org/640168
    Add advanced-format-512e-support role

Addressed by: https://review.openstack.org/640377
    Add no-op-firewall-nova-driver role

Addressed by: https://review.openstack.org/640410
    Add controller-token role

Addressed by: https://review.openstack.org/640417
    Add controller-ulimits role

Addressed by: https://review.openstack.org/640745
    Add ctlplane-ip-range role

Addressed by: https://review.openstack.org/640752
    Add lookup_plugins into validation-lib-utils role

Addressed by: https://review.openstack.org/640754
    Add node-disks role

Addressed by: https://review.openstack.org/640760
    Add node-health role

Addressed by: https://review.openstack.org/640761
    Add network-environment role

Addressed by: https://review.openstack.org/640774
    Rename undercloud-neutron-sanity-check role

Addressed by: https://review.openstack.org/640776
    Add default-node-count role

Addressed by: https://review.openstack.org/640786
    Add switch-vlans role

Addressed by: https://review.openstack.org/640972
    Add check-latest-minor-version role

Addressed by: https://review.openstack.org/640975
    Add check-network-gateway role

Addressed by: https://review.openstack.org/641027
    Add collect-flavors-and-verify-profiles role

Addressed by: https://review.openstack.org/641319
    Add deployment-images role

Addressed by: https://review.openstack.org/641327
    Add stack-health role

Addressed by: https://review.openstack.org/641328
    Add openstack-endpoints role

Addressed by: https://review.openstack.org/641330
    Add ironic-boot-configuration role

Addressed by: https://review.openstack.org/641412
    Add stonith-exists role

Addressed by: https://review.openstack.org/641634
    Add containerized-undercloud-docker role

Addressed by: https://review.openstack.org/636282
    Allow run_update_ansible_action run with Mistral or Ansible

Gerrit topic: https://review.openstack.org/#q,topic:bug/1625547,n,z

Addressed by: https://review.openstack.org/644603
    Make validation inputs configurable via Mistral

Addressed by: https://review.openstack.org/644580
    Add container_cli used for the Overcloud to the Inventory

Addressed by: https://review.openstack.org/649288
    Adding support for the new validation framework

Addressed by: https://review.openstack.org/649337
    [DNM] Get validations playbooks from playbook dir

Addressed by: https://review.openstack.org/651500
    Add nova-status Ansible role

Addressed by: https://review.openstack.org/651822
    [WIP] Update tripleo-validations documentation

Addressed by: https://review.openstack.org/653363
    Use 'DEFAULT_VALIDATIONS_BASEDIR' variable from constants.py

Addressed by: https://review.openstack.org/653367
    Deletes validations directory

Addressed by: https://review.openstack.org/653725
    Clean up some old roles dependencies.

Addressed by: https://review.openstack.org/653920
    Fix Jinja2's default filter usage

Gerrit topic: https://review.opendev.org/#/q/topic:bp/validation-framework

Addressed by: https://review.opendev.org/655366
    Fix Jinja2's default filter usage

Addressed by: https://review.opendev.org/655368
    Adds roles support in the script generating the validations doc.

Addressed by: https://review.opendev.org/656801
    Use 'DEFAULT_VALIDATIONS_BASEDIR' variable from constants.py

Addressed by: https://review.opendev.org/649337
    Get validations playbooks from playbooks directory

Addressed by: https://review.opendev.org/657740
    Get validations playbooks from playbooks directory

Addressed by: https://review.opendev.org/653367
    Deletes validations directory

Addressed by: https://review.opendev.org/654943
    TripleO documentation update for Validations

Addressed by: https://review.opendev.org/651500
    Add nova-status Ansible role

Addressed by: https://review.opendev.org/661963
    Add collect-flavors-and-verify-profiles role

Addressed by: https://review.opendev.org/661973
    Clean up some old roles dependencies.

Addressed by: https://review.opendev.org/663045
    Remove Neutron LBaaS

Addressed by: https://review.opendev.org/636282
    Allow run_update_ansible_action run with Mistral or Ansible

Addressed by: https://review.opendev.org/663631
    Allow run_update_ansible_action run with Mistral or Ansible

Addressed by: https://review.opendev.org/665098
    Add nova-status Ansible role

Addressed by: https://review.opendev.org/668685
    Allow run_update_ansible_action run with Mistral or Ansible

Addressed by: https://review.opendev.org/668672
    Add log_path_dir parameter

Addressed by: https://review.opendev.org/655165
    Allow running validations by name using Ansible by default.

Addressed by: https://review.opendev.org/668855
    Allow running validations by group using Ansible directly

Addressed by: https://review.opendev.org/669068
    Add log_path_dir parameter

Addressed by: https://review.opendev.org/669324
    Get all the available Ansible variables for each validations

Addressed by: https://review.opendev.org/669368
    Add support for running validations through ansible-playbook

Addressed by: https://review.opendev.org/669635
    Processing available Ansible variables as 'parameter' not as 'metadata'

Addressed by: https://review.opendev.org/669637
    Renaming 'Metadata' column by 'Parameters'

Addressed by: https://review.opendev.org/669697
    Add extra_vars parameter to run_ansible_playbook function

Addressed by: https://review.opendev.org/669814
    Allow running validations by name using Ansible by default.

Addressed by: https://review.opendev.org/669846
    Allow running validations by group using Ansible by default

Addressed by: https://review.opendev.org/670091
    [WIP] Allow running validations with custom extra_args

Addressed by: https://review.opendev.org/670352
    [WIP] Add --parameters argument to get only the variables

Addressed by: https://review.opendev.org/670825
    TripleO documentation update for Validations

Addressed by: https://review.opendev.org/670974
    Get all the available Ansible variables for each validations

Addressed by: https://review.opendev.org/670975
    Processing available Ansible variables as 'parameters' not as 'metadata'

Addressed by: https://review.opendev.org/671242
    Improve running prep validations using Ansible documentation

Addressed by: https://review.opendev.org/671289
    Add plan parameter to run_ansible_playbook function

Addressed by: https://review.opendev.org/671439
    Allow to pass "plan" name to tripleo-ansible-inventory script

Addressed by: https://review.opendev.org/671491
    Don't use local connection by default for the validations

Addressed by: https://review.opendev.org/671683
    Use static inventory YAML file for saving execution time

Addressed by: https://review.opendev.org/671983
    Renaming 'Metadata' column by 'Parameters' when listing validations

Addressed by: https://review.opendev.org/672052
    Adding new role automation with Ansible

Addressed by: https://review.opendev.org/672220
    [WIP] Add autodocumentation for roles and library

Addressed by: https://review.opendev.org/672278
    Add gathering_policy parameter to run_ansible_playbook function

Addressed by: https://review.opendev.org/673264
    Add extra_vars parameter to run_ansible_playbook function

Addressed by: https://review.opendev.org/673756
    Allow running validations with custom extra variables

Addressed by: https://review.opendev.org/673759
    Add --parameters and --create-vars-file arguments to the list subcommand

Addressed by: https://review.opendev.org/673982
    Add plan parameter to run_ansible_playbook function

Addressed by: https://review.opendev.org/673983
    Don't use local connection by default for the validations

Addressed by: https://review.opendev.org/673984
    Use static inventory YAML file for saving execution time

Addressed by: https://review.opendev.org/673985
    Add gathering_policy parameter to run_ansible_playbook function

Addressed by: https://review.opendev.org/674567
    Adding new role automation with Ansible

Addressed by: https://review.opendev.org/675291
    Exit listing validations with a proper return code

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.