Enhance logging experience with remote logs collecting and filtering

Registered by Bogdan Dobrelya on 2014-01-16

I'd like to raise this blueprint again, Kibana UI is a good option for filtering logs in RT, and Elasticsearch is a good backend as a search-engine for Logstash logging engine to replace(evntually) rsyslog as well.

We should also reuse Openstack Infra experience as well, see http://logstash.openstack.org as a good example of the logs content analysis implementation.

Goal for beta (milestone 1) is to support deployment for non clustered Elasticsearch embedded with Logstash and Kibana3 UI (also embedded) resided onto the Fuel master node. The maximum messages throughput provided would be not greater than 2k/sec.

Goal for product (milestone 2) is to provide a Logging as a Service for all Openstack projects as well for its Cloud Apps (Paas) and user instances - exposed via REST API. It should also support horizontally scaled clustered automated deployments on demand. The maximum messages throughput provided should be near to 60k/sec.

Blueprint information

Status:
Complete
Approver:
Mike Scherbakov
Priority:
Low
Drafter:
Bogdan Dobrelya
Direction:
Needs approval
Assignee:
Bogdan Dobrelya
Definition:
Superseded
Series goal:
None
Implementation:
Informational Informational
Milestone target:
None
Completed by
Bogdan Dobrelya on 2015-02-26

Related branches

Sprints

(?)

Work Items

Work items:
Design: DONE
PoC: DONE
Stability issues: DONE
Gerrit patches for verbatim copies of puppet-logstash, puppet-elasticsearch: DONE
Blueprint conversion to the Logging as a service (LogaaS): INPROGRESS

This blueprint contains Public information 
Everyone can see this information.