hook failed: "certificates-relation-changed" for vault:certificates

Bug #1810910 reported by Frode Nordahl
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Keystone Charm
Fix Released
High
Frode Nordahl

Bug Description

2019-01-08 09:10:11 INFO juju-log certificates:5: Registered config file: /etc/keystone/keystone.conf
2019-01-08 09:10:11 INFO juju-log certificates:5: Registered config file: /etc/keystone/logging.conf
2019-01-08 09:10:11 INFO juju-log certificates:5: Registered config file: /etc/haproxy/haproxy.cfg
2019-01-08 09:10:11 INFO juju-log certificates:5: Registered config file: /etc/apache2/sites-available/openstack_https_frontend.conf
2019-01-08 09:10:11 INFO juju-log certificates:5: Registered config file: /etc/keystone/policy.json
2019-01-08 09:10:11 INFO juju-log certificates:5: Registered config file: /etc/cron.d/keystone-token-flush
2019-01-08 09:10:11 INFO juju-log certificates:5: Registered config file: /etc/cron.d/keystone-fernet-rotate-sync
2019-01-08 09:10:11 INFO juju-log certificates:5: Registered config file: /etc/apache2/sites-enabled/wsgi-openstack-api.conf
2019-01-08 09:10:11 INFO juju-log certificates:5: Registered config file: /etc/memcached.conf
2019-01-08 09:10:12 INFO juju-log certificates:5: Making dir /etc/apache2/ssl/keystone root:root 555
2019-01-08 09:10:12 INFO juju-log certificates:5: Installing new CA cert at: /usr/local/share/ca-certificates/keystone_juju_ca_cert.crt
2019-01-08 09:10:12 DEBUG juju-log certificates:5: Writing file /usr/local/share/ca-certificates/keystone_juju_ca_cert.crt root:root 444
2019-01-08 09:10:12 DEBUG certificates-relation-changed Clearing symlinks in /etc/ssl/certs...
2019-01-08 09:10:13 DEBUG certificates-relation-changed done.
2019-01-08 09:10:13 DEBUG certificates-relation-changed Updating certificates in /etc/ssl/certs...
2019-01-08 09:10:14 DEBUG certificates-relation-changed 149 added, 0 removed; done.
2019-01-08 09:10:14 DEBUG certificates-relation-changed Running hooks in /etc/ca-certificates/update.d...
2019-01-08 09:10:14 DEBUG certificates-relation-changed done.
2019-01-08 09:10:14 DEBUG juju-log certificates:5: Writing file /etc/apache2/ssl/keystone/cert_juju-d9e982-0.lxd root:root 640
2019-01-08 09:10:14 DEBUG juju-log certificates:5: Writing file /etc/apache2/ssl/keystone/key_juju-d9e982-0.lxd root:root 640
2019-01-08 09:10:15 INFO juju-log certificates:5: Loaded template from templates/ocata/logging.conf
2019-01-08 09:10:15 INFO juju-log certificates:5: Rendering from template: /etc/keystone/logging.conf
2019-01-08 09:10:15 INFO juju-log certificates:5: Wrote template /etc/keystone/logging.conf.
2019-01-08 09:10:15 INFO juju-log certificates:5: Loaded template from templates/keystone-token-flush
2019-01-08 09:10:15 INFO juju-log certificates:5: Rendering from template: /etc/cron.d/keystone-token-flush
2019-01-08 09:10:15 INFO juju-log certificates:5: Wrote template /etc/cron.d/keystone-token-flush.
2019-01-08 09:10:15 DEBUG certificates-relation-changed lxc
2019-01-08 09:10:15 INFO juju-log certificates:5: Configuring Keystone to use a random admin token.
2019-01-08 09:10:15 INFO juju-log certificates:5: Loading a previously generated admin token from /var/lib/keystone/keystone.token
2019-01-08 09:10:15 INFO juju-log certificates:5: Loaded template from templates/wsgi-openstack-api.conf
2019-01-08 09:10:15 INFO juju-log certificates:5: Rendering from template: /etc/apache2/sites-enabled/wsgi-openstack-api.conf
2019-01-08 09:10:15 INFO juju-log certificates:5: Wrote template /etc/apache2/sites-enabled/wsgi-openstack-api.conf.
2019-01-08 09:10:15 DEBUG juju-log certificates:5: Ensuring haproxy enabled in /etc/default/haproxy.
2019-01-08 09:10:16 INFO juju-log certificates:5: HAProxy context is incomplete, this unit has no peers.
2019-01-08 09:10:16 INFO juju-log certificates:5: Loaded template from /var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/templates/haproxy.cfg
2019-01-08 09:10:16 INFO juju-log certificates:5: Rendering from template: /etc/haproxy/haproxy.cfg
2019-01-08 09:10:16 INFO juju-log certificates:5: Wrote template /etc/haproxy/haproxy.cfg.
2019-01-08 09:10:16 DEBUG certificates-relation-changed Considering dependency setenvif for ssl:
2019-01-08 09:10:16 DEBUG certificates-relation-changed Module setenvif already enabled
2019-01-08 09:10:16 DEBUG certificates-relation-changed Considering dependency mime for ssl:
2019-01-08 09:10:16 DEBUG certificates-relation-changed Module mime already enabled
2019-01-08 09:10:16 DEBUG certificates-relation-changed Considering dependency socache_shmcb for ssl:
2019-01-08 09:10:16 DEBUG certificates-relation-changed Module socache_shmcb already enabled
2019-01-08 09:10:16 DEBUG certificates-relation-changed Module ssl already enabled
2019-01-08 09:10:16 DEBUG certificates-relation-changed Module proxy already enabled
2019-01-08 09:10:16 DEBUG certificates-relation-changed Considering dependency proxy for proxy_http:
2019-01-08 09:10:16 DEBUG certificates-relation-changed Module proxy already enabled
2019-01-08 09:10:16 DEBUG certificates-relation-changed Module proxy_http already enabled
2019-01-08 09:10:16 DEBUG certificates-relation-changed Module headers already enabled
2019-01-08 09:10:16 INFO juju-log certificates:5: Loaded template from templates/openstack_https_frontend.conf
2019-01-08 09:10:16 INFO juju-log certificates:5: Rendering from template: /etc/apache2/sites-available/openstack_https_frontend.conf
2019-01-08 09:10:16 INFO juju-log certificates:5: Wrote template /etc/apache2/sites-available/openstack_https_frontend.conf.
2019-01-08 09:10:16 INFO juju-log certificates:5: Loaded template from /var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/templates/memcached.conf
2019-01-08 09:10:16 INFO juju-log certificates:5: Rendering from template: /etc/memcached.conf
2019-01-08 09:10:16 INFO juju-log certificates:5: Wrote template /etc/memcached.conf.
2019-01-08 09:10:16 INFO juju-log certificates:5: Configuring Keystone to use a random admin token.
2019-01-08 09:10:16 INFO juju-log certificates:5: Loading a previously generated admin token from /var/lib/keystone/keystone.token
2019-01-08 09:10:16 INFO juju-log certificates:5: Loaded template from templates/queens/policy.json
2019-01-08 09:10:16 INFO juju-log certificates:5: Rendering from template: /etc/keystone/policy.json
2019-01-08 09:10:16 INFO juju-log certificates:5: Wrote template /etc/keystone/policy.json.
2019-01-08 09:10:16 INFO juju-log certificates:5: Configuring Keystone to use a random admin token.
2019-01-08 09:10:17 INFO juju-log certificates:5: Loading a previously generated admin token from /var/lib/keystone/keystone.token
2019-01-08 09:10:17 INFO juju-log certificates:5: HAProxy context is incomplete, this unit has no peers.
2019-01-08 09:10:17 DEBUG certificates-relation-changed lxc
2019-01-08 09:10:17 INFO juju-log certificates:5: Loaded template from templates/ocata/keystone.conf
2019-01-08 09:10:17 INFO juju-log certificates:5: Rendering from template: /etc/keystone/keystone.conf
2019-01-08 09:10:17 INFO juju-log certificates:5: Wrote template /etc/keystone/keystone.conf.
2019-01-08 09:10:17 INFO juju-log certificates:5: Loaded template from templates/keystone-fernet-rotate-sync
2019-01-08 09:10:17 INFO juju-log certificates:5: Rendering from template: /etc/cron.d/keystone-fernet-rotate-sync
2019-01-08 09:10:17 INFO juju-log certificates:5: Wrote template /etc/cron.d/keystone-fernet-rotate-sync.
2019-01-08 09:10:17 DEBUG certificates-relation-changed lxc
2019-01-08 09:10:17 INFO juju-log certificates:5: Configuring Keystone to use a random admin token.
2019-01-08 09:10:17 INFO juju-log certificates:5: Loading a previously generated admin token from /var/lib/keystone/keystone.token
2019-01-08 09:10:17 INFO juju-log certificates:5: Configuring Keystone to use a random admin token.
2019-01-08 09:10:17 INFO juju-log certificates:5: Loading a previously generated admin token from /var/lib/keystone/keystone.token
2019-01-08 09:10:18 DEBUG certificates-relation-changed Site openstack_https_frontend already enabled
2019-01-08 09:10:19 DEBUG juju-log certificates:5: Database is initialised
2019-01-08 09:10:19 DEBUG certificates-relation-changed /usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py:179: UserWarning: Using keystoneclient sessions has been deprecated. Please update your software to use keystoneauth1.
2019-01-08 09:10:19 DEBUG certificates-relation-changed warnings.warn('Using keystoneclient sessions has been deprecated. '
2019-01-08 09:11:04 DEBUG certificates-relation-changed Traceback (most recent call last):
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 557, in <module>
2019-01-08 09:11:04 DEBUG certificates-relation-changed admin_token=spec['admin_token'])
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 487, in get_manager
2019-01-08 09:11:04 DEBUG certificates-relation-changed api_local_endpoint, admin_token, api_version)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 102, in _retry_on_exception_inner_2
2019-01-08 09:11:04 DEBUG certificates-relation-changed return f(*args, **kwargs)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 150, in get_keystone_manager
2019-01-08 09:11:04 DEBUG certificates-relation-changed for svc in manager.api.services.list():
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/services.py", line 93, in list
2019-01-08 09:11:04 DEBUG certificates-relation-changed **kwargs)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 75, in func
2019-01-08 09:11:04 DEBUG certificates-relation-changed return f(*args, **new_kwargs)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 397, in list
2019-01-08 09:11:04 DEBUG certificates-relation-changed self.collection_key)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 125, in _list
2019-01-08 09:11:04 DEBUG certificates-relation-changed resp, body = self.client.get(url, **kwargs)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 304, in get
2019-01-08 09:11:04 DEBUG certificates-relation-changed return self.request(url, 'GET', **kwargs)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 463, in request
2019-01-08 09:11:04 DEBUG certificates-relation-changed resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request
2019-01-08 09:11:04 DEBUG certificates-relation-changed return self.session.request(url, method, **kwargs)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 428, in request
2019-01-08 09:11:04 DEBUG certificates-relation-changed resp = send(**kwargs)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 472, in _send_request
2019-01-08 09:11:04 DEBUG certificates-relation-changed raise exceptions.ConnectionRefused(msg)
2019-01-08 09:11:04 DEBUG certificates-relation-changed ConnectFailure: Unable to establish connection to http://localhost:35337/v3/services?
2019-01-08 09:11:04 ERROR juju-log certificates:5: The call within manager.py failed with the error: 'Unable to establish connection to http://localhost:35337/v3/services?'. The call was: path=['resolve_domain_id'], args=('default',), kwargs={}, api_version=None
2019-01-08 09:11:04 INFO juju-log certificates:5: Retrying '_ensure_initial_admin' 3 more times (delay=3)
2019-01-08 09:11:52 DEBUG certificates-relation-changed Traceback (most recent call last):
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 557, in <module>
2019-01-08 09:11:52 DEBUG certificates-relation-changed admin_token=spec['admin_token'])
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 487, in get_manager
2019-01-08 09:11:52 DEBUG certificates-relation-changed api_local_endpoint, admin_token, api_version)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 102, in _retry_on_exception_inner_2
2019-01-08 09:11:52 DEBUG certificates-relation-changed return f(*args, **kwargs)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 150, in get_keystone_manager
2019-01-08 09:11:52 DEBUG certificates-relation-changed for svc in manager.api.services.list():
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/services.py", line 93, in list
2019-01-08 09:11:52 DEBUG certificates-relation-changed **kwargs)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 75, in func
2019-01-08 09:11:52 DEBUG certificates-relation-changed return f(*args, **new_kwargs)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 397, in list
2019-01-08 09:11:52 DEBUG certificates-relation-changed self.collection_key)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 125, in _list
2019-01-08 09:11:52 DEBUG certificates-relation-changed resp, body = self.client.get(url, **kwargs)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 304, in get
2019-01-08 09:11:52 DEBUG certificates-relation-changed return self.request(url, 'GET', **kwargs)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 463, in request
2019-01-08 09:11:52 DEBUG certificates-relation-changed resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request
2019-01-08 09:11:52 DEBUG certificates-relation-changed return self.session.request(url, method, **kwargs)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 428, in request
2019-01-08 09:11:52 DEBUG certificates-relation-changed resp = send(**kwargs)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 472, in _send_request
2019-01-08 09:11:52 DEBUG certificates-relation-changed raise exceptions.ConnectionRefused(msg)
2019-01-08 09:11:52 DEBUG certificates-relation-changed ConnectFailure: Unable to establish connection to http://localhost:35337/v3/services?
2019-01-08 09:11:53 ERROR juju-log certificates:5: The call within manager.py failed with the error: 'Unable to establish connection to http://localhost:35337/v3/services?'. The call was: path=['resolve_domain_id'], args=('default',), kwargs={}, api_version=None
2019-01-08 09:11:53 INFO juju-log certificates:5: Retrying '_ensure_initial_admin' 2 more times (delay=6)
2019-01-08 09:12:44 DEBUG certificates-relation-changed Traceback (most recent call last):
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 557, in <module>
2019-01-08 09:12:44 DEBUG certificates-relation-changed admin_token=spec['admin_token'])
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 487, in get_manager
2019-01-08 09:12:44 DEBUG certificates-relation-changed api_local_endpoint, admin_token, api_version)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 102, in _retry_on_exception_inner_2
2019-01-08 09:12:44 DEBUG certificates-relation-changed return f(*args, **kwargs)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 150, in get_keystone_manager
2019-01-08 09:12:44 DEBUG certificates-relation-changed for svc in manager.api.services.list():
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/services.py", line 93, in list
2019-01-08 09:12:44 DEBUG certificates-relation-changed **kwargs)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 75, in func
2019-01-08 09:12:44 DEBUG certificates-relation-changed return f(*args, **new_kwargs)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 397, in list
2019-01-08 09:12:44 DEBUG certificates-relation-changed self.collection_key)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 125, in _list
2019-01-08 09:12:44 DEBUG certificates-relation-changed resp, body = self.client.get(url, **kwargs)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 304, in get
2019-01-08 09:12:44 DEBUG certificates-relation-changed return self.request(url, 'GET', **kwargs)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 463, in request
2019-01-08 09:12:44 DEBUG certificates-relation-changed resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request
2019-01-08 09:12:44 DEBUG certificates-relation-changed return self.session.request(url, method, **kwargs)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 428, in request
2019-01-08 09:12:44 DEBUG certificates-relation-changed resp = send(**kwargs)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 472, in _send_request
2019-01-08 09:12:44 DEBUG certificates-relation-changed raise exceptions.ConnectionRefused(msg)
2019-01-08 09:12:44 DEBUG certificates-relation-changed ConnectFailure: Unable to establish connection to http://localhost:35337/v3/services?
2019-01-08 09:12:44 ERROR juju-log certificates:5: The call within manager.py failed with the error: 'Unable to establish connection to http://localhost:35337/v3/services?'. The call was: path=['resolve_domain_id'], args=('default',), kwargs={}, api_version=None
2019-01-08 09:12:44 INFO juju-log certificates:5: Retrying '_ensure_initial_admin' 1 more times (delay=9)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Traceback (most recent call last):
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 557, in <module>
2019-01-08 09:13:38 DEBUG certificates-relation-changed admin_token=spec['admin_token'])
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 487, in get_manager
2019-01-08 09:13:38 DEBUG certificates-relation-changed api_local_endpoint, admin_token, api_version)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 102, in _retry_on_exception_inner_2
2019-01-08 09:13:38 DEBUG certificates-relation-changed return f(*args, **kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 150, in get_keystone_manager
2019-01-08 09:13:38 DEBUG certificates-relation-changed for svc in manager.api.services.list():
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/services.py", line 93, in list
2019-01-08 09:13:38 DEBUG certificates-relation-changed **kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 75, in func
2019-01-08 09:13:38 DEBUG certificates-relation-changed return f(*args, **new_kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 397, in list
2019-01-08 09:13:38 DEBUG certificates-relation-changed self.collection_key)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 125, in _list
2019-01-08 09:13:38 DEBUG certificates-relation-changed resp, body = self.client.get(url, **kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 304, in get
2019-01-08 09:13:38 DEBUG certificates-relation-changed return self.request(url, 'GET', **kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 463, in request
2019-01-08 09:13:38 DEBUG certificates-relation-changed resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request
2019-01-08 09:13:38 DEBUG certificates-relation-changed return self.session.request(url, method, **kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 428, in request
2019-01-08 09:13:38 DEBUG certificates-relation-changed resp = send(**kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 472, in _send_request
2019-01-08 09:13:38 DEBUG certificates-relation-changed raise exceptions.ConnectionRefused(msg)
2019-01-08 09:13:38 DEBUG certificates-relation-changed ConnectFailure: Unable to establish connection to http://localhost:35337/v3/services?
2019-01-08 09:13:38 ERROR juju-log certificates:5: The call within manager.py failed with the error: 'Unable to establish connection to http://localhost:35337/v3/services?'. The call was: path=['resolve_domain_id'], args=('default',), kwargs={}, api_version=None
2019-01-08 09:13:38 DEBUG certificates-relation-changed Traceback (most recent call last):
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/certificates-relation-changed", line 882, in <module>
2019-01-08 09:13:38 DEBUG certificates-relation-changed main()
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/certificates-relation-changed", line 875, in main
2019-01-08 09:13:38 DEBUG certificates-relation-changed hooks.execute(sys.argv)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/hookenv.py", line 909, in execute
2019-01-08 09:13:38 DEBUG certificates-relation-changed self._hooks[hook_name]()
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/utils.py", line 1479, in wrapped_f
2019-01-08 09:13:38 DEBUG certificates-relation-changed stopstart, restart_functions)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/host.py", line 739, in restart_on_change_helper
2019-01-08 09:13:38 DEBUG certificates-relation-changed r = lambda_f()
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/utils.py", line 1478, in <lambda>
2019-01-08 09:13:38 DEBUG certificates-relation-changed (lambda: f(*args, **kwargs)), __restart_map_cache['cache'],
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/certificates-relation-changed", line 854, in certs_changed
2019-01-08 09:13:38 DEBUG certificates-relation-changed ensure_initial_admin(config)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1359, in ensure_initial_admin
2019-01-08 09:13:38 DEBUG certificates-relation-changed return _ensure_initial_admin(config)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2019-01-08 09:13:38 DEBUG certificates-relation-changed return f(*args, **kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1303, in _ensure_initial_admin
2019-01-08 09:13:38 DEBUG certificates-relation-changed default_domain_id = create_or_show_domain(DEFAULT_DOMAIN)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 953, in create_or_show_domain
2019-01-08 09:13:38 DEBUG certificates-relation-changed domain_id = manager.resolve_domain_id(name)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1019, in __call__
2019-01-08 09:13:38 DEBUG certificates-relation-changed return _proxy_manager_call(self._path, self.api_version, args, kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1052, in _proxy_manager_call
2019-01-08 09:13:38 DEBUG certificates-relation-changed raise e
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1049, in _proxy_manager_call
2019-01-08 09:13:38 DEBUG certificates-relation-changed raise RuntimeError(s)
2019-01-08 09:13:38 DEBUG certificates-relation-changed RuntimeError: The call within manager.py failed with the error: 'Unable to establish connection to http://localhost:35337/v3/services?'. The call was: path=['resolve_domain_id'], args=('default',), kwargs={}, api_version=None
2019-01-08 09:13:38 DEBUG certificates-relation-changed Traceback (most recent call last):
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 553, in <module>
2019-01-08 09:13:38 DEBUG certificates-relation-changed spec = json.loads(data)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
2019-01-08 09:13:38 DEBUG certificates-relation-changed return _default_decoder.decode(s)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
2019-01-08 09:13:38 DEBUG certificates-relation-changed obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2019-01-08 09:13:38 DEBUG certificates-relation-changed TypeError: expected string or buffer
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed /var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py: something went wrong: Unable to establish connection to http://localhost:35337/v3/services?
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed /var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py: something went wrong: Unable to establish connection to http://localhost:35337/v3/services?
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed /var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py: something went wrong: Unable to establish connection to http://localhost:35337/v3/services?
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed /var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py: something went wrong: Unable to establish connection to http://localhost:35337/v3/services?
2019-01-08 09:13:38 DEBUG certificates-relation-changed /var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py: something went wrong: expected string or buffer
2019-01-08 09:13:38 DEBUG certificates-relation-changed Traceback (most recent call last):
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 585, in <module>
2019-01-08 09:13:38 DEBUG certificates-relation-changed uds_client.send(result_json)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/uds_comms.py", line 227, in send
2019-01-08 09:13:38 DEBUG certificates-relation-changed raise UDSException(str(e))
2019-01-08 09:13:38 DEBUG certificates-relation-changed uds_comms.UDSException: [Errno 32] Broken pipe
2019-01-08 09:13:38 ERROR juju.worker.uniter.operation runhook.go:132 hook "certificates-relation-changed" failed: exit status 1

Revision history for this message
Frode Nordahl (fnordahl) wrote :

# cat /var/log/apache2/error.log
[Tue Jan 08 09:10:19.337478 2019] [ssl:emerg] [pid 17268:tid 140175730857856] AH01903: Failed to configure CA certificate chain!
[Tue Jan 08 09:10:19.337549 2019] [ssl:emerg] [pid 17268:tid 140175730857856] AH02312: Fatal error initialising mod_ssl, exiting.
AH00016: Configuration Failed

# cat /etc/apache2/ssl/keystone/cert_10.219.3.172
[ snip ]
w5m5YwDz0BoWXHt3saHMllfd1rDNpu+UvZVrqAEKBVXha+iV4XBZP+1GxjitpclG
ny7J+kEk
-----END CERTIFICATE----------BEGIN CERTIFICATE-----
MIIDADCCAeigAwIBAgIUL1jUPL/4OJ9hGr/9KfSvCXErLncwDQYJKoZIhvcNAQEL
[ snip ]

Looking at the above it appears a separator between the host certificate and the chain certificate is missing.

I confirmed this to be the case by manually editing the file and adding a newline between the end/begin certificate markers.

This occurs with Keystone charm from latest stable and latest master.

Revision history for this message
Frode Nordahl (fnordahl) wrote :

This can be validated using the charm-vault func-smoke test

Changed in charm-keystone:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Frode Nordahl (fnordahl)
milestone: none → 19.04
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/629129

Changed in charm-keystone:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-keystone (master)

Reviewed: https://review.openstack.org/629129
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=8fff100a0da559238563dc1070477f1ebd5a136c
Submitter: Zuul
Branch: master

commit 8fff100a0da559238563dc1070477f1ebd5a136c
Author: Frode Nordahl <email address hidden>
Date: Tue Jan 8 11:00:40 2019 +0100

    Sync charm-helpers to fix certificate separation

    Change-Id: I0af42ea9642f3bd9181d729d73219753621e29be
    Closes-Bug: #1810910

Changed in charm-keystone:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-keystone (stable/18.11)

Fix proposed to branch: stable/18.11
Review: https://review.openstack.org/629254

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-keystone (stable/18.11)

Reviewed: https://review.openstack.org/629254
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=6d4ba63814a0cd3c486702187702230fde6c31d7
Submitter: Zuul
Branch: stable/18.11

commit 6d4ba63814a0cd3c486702187702230fde6c31d7
Author: Frode Nordahl <email address hidden>
Date: Tue Jan 8 11:00:40 2019 +0100

    Sync charm-helpers to fix certificate separation

    Change-Id: I0af42ea9642f3bd9181d729d73219753621e29be
    Closes-Bug: #1810910
    (cherry picked from commit 8fff100a0da559238563dc1070477f1ebd5a136c)

David Ames (thedac)
Changed in charm-keystone:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.