Merge OpenStack Puppet Modules

Registered by Matthew Mosesohn on 2013-11-21

In order to achieve better stability in Fuel and benefit the community, we need to merge our Puppet modules upstream. This process involves merging core OpenStack components first, then supporting components/HA architecture second.

Feature Lead: Aleksandr Didenko
Mandatory Design Reviewers: Vladimir Kuklin
Developers: Aleksandr DIdenko, Bogdan Dobrelya
QA: Anastasia Urlapova
Acceptance criteria: system tests passed

Whiteboard

[Dmitry Borodaenko 2014-12-16]
Closed as Implemented in 6.0: the only way to *complete* a blueprint defined like this one is to implement a CI process that would keep Fuel in sync with all upstream modules *automatically*. Until we can do that, best we can do is sync a specific set of modules up to specific versions in every release of Fuel, and the only way to *track* that kind of work is to create a separate bug for every such update, per module. Since there's no need for design, it would be easier to track that as bugs instead of blueprints.

Early components:
- Nova
- Glance
- Cinder
- Swift
Later (harder) components:
- Keystone
- Horizon
- Neutron
- Savanna
- Murano
- Heat
- Ceph (custom, no direct upstream. Needs to be moved (Xarses)
Supporting components:
- MySQL
- Galera
- Cobbler
- RabbitMQ
- QPID
- Corosync/Pacemaker
- HAProxy
- PostgreSQL

---------------------
Testing methodology:
https://etherpad.openstack.org/p/fuel_puppet_modules_upgrade

---------------------
Gerrit topic: https://review.openstack.org/#q,topic:bp/merge-openstack-puppet-modules,n,z

---------------------
MERGED:

* Essential patches needed for proper CI and testing of OpenStack modules synced from upstream:

Addressed by: https://review.openstack.org/99902
    Sync puppetlabs-stdlib module to 4.2.1

Addressed by: https://review.openstack.org/99904
    Remove stdlib::safe_package dependencies

Addressed by: https://review.openstack.org/99946
    Adapt keystone_endpoint for new upstream modules

Addressed by: https://review.openstack.org/100821
    Refactor mysql for upstream modules

* Openstack modules:

Addressed by: https://review.openstack.org/101166
    Sync cinder module from puppetlabs upstream

Addressed by: https://review.openstack.org/101471
    Adapt synced puppetlab-cinder for Fuel usage

Addressed by: https://review.openstack.org/86350
    Sync puppet-nova

Addressed by: https://review.openstack.org/86351
    Adapt synced puppet-nova for Fuel usage

Addressed by: https://review.openstack.org/86574
    Sync puppet-heat

Addressed by: https://review.openstack.org/86598
    Adapt synced puppet-heat for Fuel usage

Addressed by: https://review.openstack.org/99110
    Sync glance module from puppetlabs upstream

Addressed by: https://review.openstack.org/99150
    Adapt synced puppetlab-glance for Fuel usage

Addressed by: https://review.openstack.org/103078
    Sync puppet-ceilometer

Addressed by: https://review.openstack.org/103092
    Adapt synced puppet-ceilometer for Fuel usage (WIP)

Addressed by: https://review.openstack.org/127268
    Syn swift module with upstream

Addressed by: https://review.openstack.org/127589
    Adapt synced swift module

Addressed by: https://review.openstack.org/86007
    Sync puppet-keystone

Addressed by: https://review.openstack.org/86008
    Adapt synced puppet-keystone for Fuel usage

Addressed by: https://review.openstack.org/103277
    *WIP* Purge neutron folder before sync

Addressed by: https://review.openstack.org/103278
    Sync with upstream puppet-neutron

* Non-openstack modules:

Addressed by: https://review.openstack.org/92351
    Sync firewall module from puppetlabs upstream

Addressed by: https://review.openstack.org/92361
    Adapt synced puppet-firewall for Fuel usage

Addressed by: https://review.openstack.org/99966
    Sync puppetlabs-apt to 1.4.2

Addressed by: https://review.openstack.org/100850
    Sync puppet ssh module to v2.3.6 from upstream

Addressed by: https://review.openstack.org/100878
    Remove unused puppet module mmm

Addressed by: https://review.openstack.org/100937
    Remove unused puppet module nagios

Addressed by: https://review.openstack.org/100947
    Remove unnecessary files from puppet/astute directory

Addressed by: https://review.openstack.org/101167
    Remove unused puppet module squid

Addressed by: https://review.openstack.org/101215
    Sync puppet concat module to v1.1.0 from upstream

Addressed by: https://review.openstack.org/101229
    Remove unused puppet module git

Addressed by: https://review.openstack.org/101550
    Remove unused module keepalived

Addressed by: https://review.openstack.org/102138
    Remove unused puppet module selinux

Addressed by: https://review.openstack.org/102516
    Remove unused puppet module puppetdb

Addressed by: https://review.openstack.org/102184
    Sync puppet module lvm to v0.3.1 from upstream

Addressed by: https://review.openstack.org/102231
    Sync puppet module memcached to v2.5.0 from upstream

Addressed by: https://review.openstack.org/101198
    Sync sysctl module to v0.0.4 from upstream

Addressed by: https://review.openstack.org/101202
    Adapt synced sysctl module for Fuel

---------------------
NOT MERGED:

* Openstack modules:

Addressed by: https://review.openstack.org/132687
    Add new module 'apache'

Addressed by: https://review.openstack.org/132564
    Sync horizon module with upstream

Addressed by: https://review.openstack.org/132688
    Adapt synced horizon module

* Non-openstack modules:

Addressed by: https://review.openstack.org/102483
    Sync puppet module ntp to v3.1.1 from upstream

Addressed by: https://review.openstack.org/102830
    Adapt puppet module ntp for Fuel

Addressed by: https://review.openstack.org/102871
    Sync puppet module rsync to v0.3.0 from upstream

Addressed by: https://review.openstack.org/103075
    Sync puppet module xinetd to v1.3.0 from upstream

Addressed by: https://review.openstack.org/103655
    Clean up haproxy_service['mysqld']->Anchor['galera'] dep.

Addressed by: https://review.openstack.org/104513
    Add libguestfs-tools for Ubuntu

Addressed by: https://review.openstack.org/104607
    Revert "Sync sysctl module to v0.0.4 from upstream"

Addressed by: https://review.openstack.org/104613
    Revert "Adapt synced sysctl module for Fuel"

Addressed by: https://review.openstack.org/107638
    Sync haproxy 0.5.0 module from puppetslab.

Addressed by: https://review.openstack.org/107639
    Adapt synced haproxy 0.5.0 for Fuel usage.

Addressed by: https://review.openstack.org/119076
    Sync inifile with upstream

Gerrit topic: https://review.openstack.org/#q,topic:bp/ml2-neutron,n,z

Addressed by: https://review.openstack.org/126006
    Sync puppet-corosync

Addressed by: https://review.openstack.org/126010
    fix incompatibles between new Neutron and old Keystone manifests

Addressed by: https://review.openstack.org/126027
    Adapt synced Corosync module

Addressed by: https://review.openstack.org/126414
    Adaptation cluster::corosync::cs_service to asymetric cluster

Addressed by: https://review.openstack.org/127165
    Sync puppetlabs rabbitmq module

Addressed by: https://review.openstack.org/127166
    Adapt rabbitmq module for Fuel

Addressed by: https://review.openstack.org/128214
    Fix NSX plugin manifests for upstream neutron modules

Addressed by: https://review.openstack.org/128219
    Adding l3_ext network type for NSX support

Addressed by: https://review.openstack.org/130476
    Sync with upstream puppet-neutron

Addressed by: https://review.openstack.org/130477
    Adapt upstream puppet-neutron into fuel

Addressed by: https://review.openstack.org/130478
    Neutron HA composition model

Addressed by: https://review.openstack.org/130479
    TEMPORARY neutron-keystone fix

Addressed by: https://review.openstack.org/130484
    Adding l3_ext network type for NSX support

Addressed by: https://review.openstack.org/130723
    Adding l3_ext network type for NSX support

Addressed by: https://review.openstack.org/130739
    Fix NSX plugin for upstream neutron modules

Gerrit topic: https://review.openstack.org/#q,topic:adapt-postgres,n,z

Addressed by: https://review.openstack.org/131073
    Rebase postgresql to puppetlabs-postgresql-4.0.0

Gerrit topic: https://review.openstack.org/#q,topic:bug/1384149,n,z

Gerrit topic: https://review.openstack.org/#q,topic:bug/1388049,n,z

Addressed by: https://review.openstack.org/132217
    Fix ntp conf template name in host-upgrade.pp

Addressed by: https://review.openstack.org/129925
    Use separate pid files for Pacemaker OCF

Addressed by: https://review.openstack.org/130823
    Remove old basic service_disables

Addressed by: https://review.openstack.org/132318
    Refactor Heat HA to wrappers

Addressed by: https://review.openstack.org/129351
    New service provider

Addressed by: https://review.openstack.org/141325
    Rebase postgresql to puppetlabs-postgresql-4.0.0

Addressed by: https://review.openstack.org/145788
    Sync apt module from puppetlabs

Gerrit topic: https://review.openstack.org/#q,topic:bp/iotrowski,n,z

Addressed by: https://review.openstack.org/146394
    Merge camptocamp/openssl upstream module

Addressed by: https://review.openstack.org/146408
    Merge puppetlabs-xinetd v1.3.1 module

Gerrit topic: https://review.openstack.org/#q,topic:bp/merge-openstack-puppet-module,n,z

Addressed by: https://review.openstack.org/148894
    Merge puppetlabs/cinder upstream module

Addressed by: https://review.openstack.org/148895
    Adapt puppetlabs/cinder module to Fuel

Addressed by: https://review.openstack.org/148916
    Initial import of puppet-openstacklib module

Addressed by: https://review.openstack.org/189281
    Merge puppetlabs/stdlib 4.6.0

Addressed by: https://review.openstack.org/189567
    Merge puppetlabs/concat 2.0.1

Gerrit topic: https://review.openstack.org/#q,topic:merge-openstack-puppet-module,n,z

Addressed by: https://review.openstack.org/189598
    Adapt tests and modules to concat 2.0.1

Addressed by: https://review.openstack.org/189689
    Merge saz/ssh 2.8.1

Addressed by: https://review.openstack.org/190055
    Merge puppetlabs/firewall 1.6.0

Addressed by: https://review.openstack.org/190083
    Merge puppetlabs/vcsrepo 1.6.0

Addressed by: https://review.openstack.org/190105
    Merge puppetlabs/inifile 1.3.0

Addressed by: https://review.openstack.org/190165
    Merge puppetlabs/apt 2.0.1

Addressed by: https://review.openstack.org/190168
    Remove unused epel and openstack_extras modules

Addressed by: https://review.openstack.org/190170
    Merge puppetlabs/lvm 0.5.0

Addressed by: https://review.openstack.org/190192
    Merge saz/memcached 2.8.1

Addressed by: https://review.openstack.org/190203
    Upgrade OpenStack puppet modules in fuel-library

Gerrit topic: https://review.openstack.org/#q,topic:bp/pacemaker-improvements,n,z

Addressed by: https://review.openstack.org/191769
    Remove openstack_extras module

Gerrit topic: https://review.openstack.org/#q,topic:bug/1383639,n,z

Addressed by: https://review.openstack.org/192597
    Adapt synced puppetlabs-mongodb module for Fuel usage

Addressed by: https://review.openstack.org/192714
    Adapt synced Heat module for Fuel usage

Addressed by: https://review.openstack.org/194091
    Adapt synced Cinder module for Fuel usage

Addressed by: https://review.openstack.org/194647
    Adapt synced Nova module for Fuel usage

Addressed by: https://review.openstack.org/195137
    Adapt synced Cinder module for Fuel usage

Addressed by: https://review.openstack.org/194683
    [DONOTMERGE] Adapt synced keystone module

Addressed by: https://review.openstack.org/192602
    Upgrade Heat puppet module

Addressed by: https://review.openstack.org/194202
    sync keystone with upstream

Gerrit topic: https://review.openstack.org/#q,topic:bug/1468230,n,z

Addressed by: https://review.openstack.org/195112
    Sync cinder with upstream

Addressed by: https://review.openstack.org/196933
    Remove puppet-network

Gerrit topic: https://review.openstack.org/#q,topic:bp/upgrade-openstack-puppet-modules,n,z

(?)

Work Items

Work items:
Design workflow doc: DONE
Essential compatibility patches: DONE
Dependency components: DONE
Removing unneeded components: DONE
Early components: DONE
Later components (only horizon left): INPROGRESS
Supporting components: TODO

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.