Granular deployment execution based on plugable tasks

Registered by Dima Shulyak on 2014-08-28

Mandatory Design Reviewers: Vladimir Kuklin, Dmitry Ilyin, Evgeniy Li, Vladimir Sharshov
Optional Reviewers: Nikolay Markov, Alexander Kislytskiy
QA: To be assigned
Implementation: Dmitry Shulyak, Dmitry Ilyin, Vladimir Sharshov

Iteration 1:

- Plugable tasks implementation in nailgun
- Astute executor that will run tasks and poll results in linear fasion for each role
- API that will allow to specify which tasks user wants to run

Iteration 2:

Mistral v2 api integration - (dshulyak) i think we will cover this with separate blueprint, so this is just fyi

Blueprint information

Status:
Complete
Approver:
Dmitry Pyzhov
Priority:
Essential
Drafter:
Dima Shulyak
Direction:
Approved
Assignee:
Dima Shulyak
Definition:
Approved
Series goal:
Accepted for 6.1.x
Implementation:
Implemented
Milestone target:
milestone icon 6.1
Started by
Dima Shulyak on 2014-08-28
Completed by
Dima Shulyak on 2015-04-16

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/in,n,z

Granular deployment for fuel nodes in fuel-specs
fuel-specs
https://review.openstack.org/113491

Add rpm/deb specs for python-tasklib package + task pack
fuel-main
refs/changes/83/119383/9
https://review.openstack.org/#/c/119383/

Support for plugable actions in nailgun
fuel-web
refs/changes/26/113226/16
https://review.openstack.org/#/c/113226/

Tasklib implementation in python
fuel-web
refs/changes/11/118311/23
https://review.openstack.org/#/c/118311

Granular deployment (basic version)
fuel-astute
refs/changes/17/122117/5
https://review.openstack.org/#/c/122117

Add python-tasklib to centos.ks.erb/ubuntu_package + task create
fuel-library
refs/changes/42/120042/6
https://review.openstack.org/#/c/120042/

Experimental puppet-asible module
https://review.openstack.org/123247

Gerrit topic: https://review.openstack.org/#q,topic:bp/granular-deployment-based-on-tasks,n,z

Addressed by: https://review.openstack.org/127284
    Add patching tasks

Addressed by: https://review.openstack.org/113226
    Support for plugable actions in nailgun

Addressed by: https://review.openstack.org/118311
    Tasklib implementation in python

Addressed by: https://review.openstack.org/124728
    Upload facts for each role separately

Addressed by: https://review.openstack.org/128870
    Pack and unpack tasks and roles

Gerrit topic: https://review.openstack.org/#q,topic:bp/cinder-neutron-plugins-in-fuel,n,z

Addressed by: https://review.openstack.org/122117
    Granular deployment (basic version)

Addressed by: https://review.openstack.org/120042
    Add python-tasklib to centos.ks.erb/ubuntu_packages

Addressed by: https://review.openstack.org/129869
    Revert "Add python-tasklib to centos.ks.erb/ubuntu_packages"

Addressed by: https://review.openstack.org/130286
    Improve tasklib

Addressed by: https://review.openstack.org/130823
    Remove old basic service_disables

Gerrit topic: https://review.openstack.org/#q,topic:bp/pacemaker-improvements,n,z

Addressed by: https://review.openstack.org/132034
    Add tasklib packages

Gerrit topic: https://review.openstack.org/#q,topic:bp/merge-openstack-puppet-modules,n,z

Addressed by: https://review.openstack.org/133010
    Explicitly defines mangling real service for neutron pacemaker resources.

Addressed by: https://review.openstack.org/129925
    Use separate pid files for Pacemaker OCF

Addressed by: https://review.openstack.org/132318
    Refactor Heat HA to wrappers

Addressed by: https://review.openstack.org/133064
    Fix keystone_endpoint in ceph module

Addressed by: https://review.openstack.org/129351
    New service provider

Addressed by: https://review.openstack.org/133069
    Adapt Neutron-HA manifests for works with new pacemaker service provider

Gerrit topic: https://review.openstack.org/#q,topic:138690,n,z

Addressed by: https://review.openstack.org/138690
    Graph based task serializer for deployment

Addressed by: https://review.openstack.org/140414
    Pre/post tasks serialization

Addressed by: https://review.openstack.org/140763
    Use global scope for orchestrator tasks in main stage

Addressed by: https://review.openstack.org/141320
    Draft version

Addressed by: https://review.openstack.org/141332
    Add networkx to nailgun spec

Addressed by: https://review.openstack.org/142086
    Deployment graph configuration storage in nailgun

Addressed by: https://review.openstack.org/142428
    Add commands to upload/download release deployment tasks

Addressed by: https://review.openstack.org/142454
    Validation for granular deployment configuration

Addressed by: https://review.openstack.org/142803
    Add cli actions to sync deployment configuration

Addressed by: https://review.openstack.org/143955
    Move graph creation outside of serializaer creation

Addressed by: https://review.openstack.org/144110
    Add restart_radosgw serializer

Addressed by: https://review.openstack.org/144128
    Improve dependencies format for tasks

Addressed by: https://review.openstack.org/144279
    Sync deployment tasks from fuel-library to nailgun db

Addressed by: https://review.openstack.org/144467
    Basic conditional tasks capabilities

Gerrit topic: https://review.openstack.org/#q,topic:bug/1392779,n,z

Addressed by: https://review.openstack.org/146464
    Granular deployment: tests, fixes and refactoring

Addressed by: https://review.openstack.org/146874
    Add API to skip/execute only some tasks

Addressed by: https://review.openstack.org/146890
    Add enable_quorum task in post_deployment hooks

Addressed by: https://review.openstack.org/146916
    Deploy/granular_deploy versioning based on pre/post task availability

Addressed by: https://review.openstack.org/146920
    Add commands to upload/download release deployment tasks

Addressed by: https://review.openstack.org/146921
    Add cli actions to sync deployment configuration

Addressed by: https://review.openstack.org/147167
    Add hiera and netconfig tasks into current deployment

Addressed by: https://review.openstack.org/147794
    Bug fixes for empty tasks and shell default cwd:

Addressed by: https://review.openstack.org/147823
    Commands to execute only part of graph

Addressed by: https://review.openstack.org/148322
    Add tasks synchronization to deployment testing

Addressed by: https://review.openstack.org/148355
    Reboot tasks for pre/post deployment tasks

Addressed by: https://review.openstack.org/148579
    Separate hooks for classic and granular methods

Addressed by: https://review.openstack.org/148667
    Add ability to specify endpoint of the graph

Addressed by: https://review.openstack.org/149210
    Migrate generation adn upload keys tasks

Addressed by: https://review.openstack.org/149234
    Added upload_nodes_info post deployment hook

Addressed by: https://review.openstack.org/149316
    Add pre/post stages into graph stored in library

Addressed by: https://review.openstack.org/149530
    Remove UpdateClusterHostsInfo for granular deploy

Addressed by: https://review.openstack.org/149537
    Add tasks for updating hosts info

Gerrit topic: https://review.openstack.org/#q,topic:generate-upload-keys,n,z

Addressed by: https://review.openstack.org/149544
    Add options to rsync

Addressed by: https://review.openstack.org/149543
    Move generate keys from astute

Addressed by: https://review.openstack.org/149630
    Fix stop_current_task newline typo

Addressed by: https://review.openstack.org/150071
    Commands to execute only part of graph

Addressed by: https://review.openstack.org/150384
    Always use only deploy_legacy task for cluster.mode multinode

Addressed by: https://review.openstack.org/150838
    Add start parameter for deployment graph traversal

Addressed by: https://review.openstack.org/151152
    Migrate sync_time into granular deployment taks

Addressed by: https://review.openstack.org/151164
    Added new pre deployment task for sync_time

Addressed by: https://review.openstack.org/151214
    Remove update_quorum/upload_cirros/restart_rados tasks from astute

Addressed by: https://review.openstack.org/151258
    Migrate sync-time from astute into task

Gerrit topic: https://review.openstack.org/#q,topic:split_controllers,n,z

Addressed by: https://review.openstack.org/151209
    Split primary-controller/controller mapping logic

Gerrit topic: https://review.openstack.org/#q,topic:fix_requirements,n,z

Addressed by: https://review.openstack.org/151597
    Fix yaml dependency for tasks validator

Addressed by: https://review.openstack.org/152088
    Add start parameter for tasks execution

Addressed by: https://review.openstack.org/152096
    Always replace tasks block in deployment info

Addressed by: https://review.openstack.org/152144
    Added new hook to copy files

Addressed by: https://review.openstack.org/152434
    Add graph visualization handler

Addressed by: https://review.openstack.org/152449
    Incorrect logic of verification in graph_serializer tests

Addressed by: https://review.openstack.org/152456
    Add pydot requirements to nailgun

Addressed by: https://review.openstack.org/153483
    Add restart rados gateway task to post_deployment stage

Addressed by: https://review.openstack.org/153515
    Use unix pattern when syncing deployment tasks

Addressed by: https://review.openstack.org/154428
    Sync deployment tasks for installed releases

Addressed by: https://review.openstack.org/154458
    Add CLI for graph visualization

Addressed by: https://review.openstack.org/154818
    Use unix pattern to collect tasks for deployment

Addressed by: https://review.openstack.org/154828
    Use all dependencies during groups traversal

Addressed by: https://review.openstack.org/155209
    Correct orchestrator_task_types

Addressed by: https://review.openstack.org/155239
    Add rsync manifests/upload repos tasks in configuration

Addressed by: https://review.openstack.org/155240
    Remove rsync puppet, update repos and rsync tasks

Addressed by: https://review.openstack.org/155334
    Overwrite default /etc/apt/sources.list

Addressed by: https://review.openstack.org/155684
    Move openssl and ssh keys during upgrade

Addressed by: https://review.openstack.org/155719
    Standardize keys dir

Addressed by: https://review.openstack.org/155755
    Make one joined graph from 3 separate stages

Addressed by: https://review.openstack.org/156568
    Operate on joined graph in nailgun

Addressed by: https://review.openstack.org/156911
    Added clear_nodes_info pre-deploy task

Addressed by: https://review.openstack.org/156963
    Remove empty GranularPostDeploymentActions

Addressed by: https://review.openstack.org/157727
    Remove migrated actions

Gerrit topic: https://review.openstack.org/#q,topic:version_fix,n,z

Addressed by: https://review.openstack.org/159796
    Fix package version when using pbr

Gerrit topic: https://review.openstack.org/#q,topic:tox_and_run_tests,n,z

Addressed by: https://review.openstack.org/159108
    Added tox and run_tests.sh

Addressed by: https://review.openstack.org/162982
    Add pydot-ng as nailgun requirement

Addressed by: https://review.openstack.org/161653
    Cover all roles entities with schema

Gerrit topic: https://review.openstack.org/#q,topic:graph_render_docs,n,z

Addressed by: https://review.openstack.org/166802
    Add docs about rendering graph with fuelclient

Gerrit topic: https://review.openstack.org/#q,topic:bug/1415520,n,z

Addressed by: https://review.openstack.org/182612
    Release Notes 6.1 -- New Features (Fuel)

Gerrit topic: https://review.openstack.org/#q,topic:bug/1415520-patch8,n,z

Gerrit topic: https://review.openstack.org/#q,topic:bug/1467817,n,z

Addressed by: https://review.openstack.org/194961
    Merge branch 'master' into stable/6.1

Gerrit topic: https://review.openstack.org/#q,topic:replace_tasks,n,z

(?)

Work Items

Work items:
prepare package for Python tasklib: DONE
task api for nailgun : DONE
astute executions based on provided tasks : DONE
add yaml configuration to library puppets : DONE
add packaged tasklib to bootstrap/centos/ubuntu : DONE
implement tasklib in python : DONE
intergation testing and bugfixing: INPROGRESS

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.