Multiple External Ceph Support

This blueprint proposes that TripleO get the ability to configure a single overcloud to use an arbitrary number of external ceph clusters.

TripleO can currently configure the overcloud to use only one external ceph cluster via the environments/ceph-ansible/ceph-ansible-external.yaml [1] environment file. It is incumbent on the external Ceph cluster administrator to create a keyring [2] which can access a pool and to then provide the key for keyring, the list of Ceph monitor IP addresses, and the Ceph cluster FSID. The TripleO user may then use these parameters as input to the ceph-ansible-external.yaml [1] environment file and then start the overcloud deployment. This results parameters getting passed to the ceph-ansible client role and a ceph configuration file being created in /etc/ceph/_cluster_name_.conf containing the CephClusterFSID and CephExternalMonHost and a key file [2] being created for that Ceph cluster.

If TripleO could support multiple Ceph clusters, then a new paramter like the following would be added:

CephMultiBackendsHash:
    ceph1:
      CephClusterFSID: '4b5c8c0a-ff60-454b-a1b4-9747aa737d19'
      CephClientKey: 'AQDLOh1VgEp6FRAAFzT7Zw+Y9V6JJExQAsRnRQ=='
      CephExternalMonHost: '172.16.1.7, 172.16.1.8'
    ceph2:
      CephClusterFSID: 'a7f64266-0894-4f1e-a635-d0aeaca0e993'
      CephClientKey: 'secretsecretsecretsecretsecretsecret33=='
      CephExternalMonHost: '10.1.1.66, 10.1.1.77'
      CinderRbdPoolName: cinder
    ...
    cephN:

This would then result in TripleO running ceph-ansible for the ceph-client role N times with the same parameters for each run except different parameters would be passed per run if it's included in the cluster name's hash. E.g. the FSID would be different for the cluster named ceph1. These runs would result in N keyring files getting created in the appropriate directory, N Ceph conf files like /etc/ceph/{ceph1,ceph2,...,cephN}.conf, and so on. It is then incumbent on other tools to configure different OpenStack services which use Ceph to use that backend; e.g. the cinder.conf could have entries for more than one backend and those backends could rely on the ceph client configuration implemented by this blueprint.

[1] https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/environments/ceph-ansible/ceph-ansible-external.yaml
[2] https://docs.ceph.com/docs/master/man/8/ceph-authtool/