Alarms on Logs

Registered by Witek Bedyk

There is a need to extend the current log management stream in Monasca to support alarm generation based on log entries.
The new Monasca component "monasca-log-metrics" is located parallel to the log persister and in the stream behind the log transformer. It consumes the log message from kafka and if the log message contains a severity field with severity "critical", "error" or "warning" a metric will be created. Additional criteria for log messaged can be configured by the system operator to produce further metric types. The new created metric will be stored in kafka and in the stream before the threshold engine. The threshold engine can then handle it as every other metric.

Blueprint information

Status:
Complete
Approver:
Roland Hochmuth
Priority:
High
Drafter:
Witek Bedyk
Direction:
Approved
Assignee:
Witek Bedyk
Definition:
Approved
Series goal:
None
Implementation:
Implemented
Milestone target:
None
Started by
Roland Hochmuth
Completed by
Roland Hochmuth

Related branches

Sprints

Whiteboard

Metric Structure (JSON)
------------------------------------

{
       "metric": {
           "name": "log.warning",
           "dimensions": {
               "hostname":"mini-mon",
                "service":"monitoring",
                "component":"monasca-api",
                "path":"/var/log/monasca/monasca-api.log"
           },
           "timestamp": 1421259363,
           "value": 1.0,
           "period":"-1"
       },
       "meta": {
           "tenantId": "8ada618268ec43709a2ab8eb8ea7996c",
           "region": "useast"
       },
       "creation_time": 1421259371
}

Metric Name
-------------------

The name is either "log.critical", "log.error" or "log.warning".

Metric Dimensions
---------------------------

All the dimensions from the log message must be put into the metric structure.

Metric Timestamp
---------------------------

The timestamp from the log message must be put into the metric structure. The creation time added by the log API should be good enough, otherwise the timestamp must be parsed from the original log.

Metric Value
------------------

The value is "1.0".

Period (new field)
--------------------------

The threshold engine expects periodic occurring metrics. That means within a certain time period a metric must arrive otherwise e.g. the service state will be shown as UNDETERMINED.

The concept of "deterministic alarms" has been introduced, which will never enter the UNDETERMINED state.

The concept of "locked alarms" should be introduced, which once in ALARM state will stay in that state until reset through API.

The negative period of the metric can give a hint that the alarm with that metric should be defined as "deterministic" and/or "locked".

Gerrit topic: https://review.openstack.org/#q,topic:sporadic_metric,n,z

Gerrit topic: https://review.openstack.org/#q,topic:deterministic_alarm,n,z

Addressed by: https://review.openstack.org/303349
    (Non)deterministic support for alarms

(?)

Work Items

Work items:
extend monasca-agent to send metric with 'period': TODO
implement 'deterministic alarms': DONE
implement 'locked alarms': INPROGRESS
implement 'sporadic metrics': TODO
create new metric-generator for logs: INPROGRESS

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.