Centralised overcloud logging to logstash
It would be great if overcloud service logs (Nova, Glance et al) were browseable and searchable centrally. To this end, I propose enabling services to be configured to log to syslog as well as disk. Listening to syslog on every node will be a logstash agent. On the compute nodes, this will forward the logs into RabbitMQ on the control node. From there, both syslog and forwarded messages will be sent into a local Elasticsearch. Finally, Kibana3 will run so the logs can be browsed and searched.
Blueprint information
- Status:
- Complete
- Approver:
- Robert Collins
- Priority:
- Undefined
- Drafter:
- Alexis Lee
- Direction:
- Needs approval
- Assignee:
- Alexis Lee
- Definition:
- Superseded
- Series goal:
- None
- Implementation:
-
Unknown
- Milestone target:
- None
- Started by
- Completed by
- Steven Hardy
Related branches
Related bugs
Sprints
Whiteboard
Gerrit topic: https:/
Robert:
- Please create an etherpad for this and link it from blueprint..
- sylog is quite restrictive on logging, would really prefer not to tunnel anything through it - we can pickup all logs from the current per service mechanism
Alexis:
- Added an etherpad, click "Read the full specification"
- I've changed tack a bit so that these elements just install the software, it's up to the system integrator to configure it usefully (EG add inputs, outputs, scalability)
- This means syslog is no longer mentioned by any of the patches
Addressed by: https:/
Add elasticsearch element
Addressed by: https:/
Add logstash element
Addressed by: https:/
Add beaver element
Addressed by: https:/
Kibana element - Logstash Web UI
Cristian:
- Could it be useful to include logstash configuration and filter files? With some nice regex that initially could help operators to use this out of the box? If you want I can help with this
Alexis:
- Hi Cristian, that can be done from tripleo-incubator. I'm in favour but it's not on my radar until the above patches have merged.
Addressed by: https:/
Add elasticsearch element
Addressed by: https:/
Create Fluentd element
Gerrit topic: https:/
Addressed by: https:/
Create a file for indexing it in logstash
Gerrit topic: https:/
Addressed by: https:/
Save tempest output in logstash
Gerrit topic: https:/
Addressed by: https:/
Save bootstrap server logs in logstash
Addressed by: https:/
Add log of ansible modify container role to logstash
Addressed by: https:/
Include container build error log for logstash indexing
