Add Alarm Silencing Functionality

Registered by Andrea Adams

Alarm silencing is to add the ability to mute an alarm for some given amount of time. A silence rule would be configured based on a given alarm attribute, a start time, and an end time. This will be configurable via the API using a new API resource (see alarm manager API resource blueprint for more details). Before any notification is sent out, the alarm would be checked against any silence rules. If the current time falls in the silence window, then a notification would not be sent

The reason for adding this feature is to enable the user to silence a potentially annoying alarm for an amount of time while the root cause of the alarm is investigated and/or fixed.

The needed changes for this include changes to the API, the python-monascaclient, and the Notification engine. For API changes, please see the alarm manager API resources blueprint. The Notification engine will need to be modified to check an alarm against existing silence rules. The Notification engine will start by checking an alarm against any silence, inhibition, and grouping rules. If an alarm falls inside the configured start and end time window, then no notification is sent out. Otherwise, a notification is sent. Then an alarm needs to go through any inhibition and grouping checks so it is sent to a Kafka topic for further analysis. The Alarm Manager and the Notification Engine will read rules on start up from the database, and in run-time from a Kafka topic called Alarm Manager Rules and keep the rules in memory. Please see the flow diagram for more information.

Please see examples in Monasca wiki page: https://wiki.openstack.org/wiki/Monasca#Alarm_Managers

Blueprint information

Status:
Not started
Approver:
None
Priority:
High
Drafter:
Andrea Adams
Direction:
Needs approval
Assignee:
Andrea Adams
Definition:
New
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/alarm-inhibition,n,z

Addressed by: https://review.openstack.org/434537
    [WIP]Modify Notification Engine to allow inhibit, silence, and group

Addressed by: https://review.openstack.org/438032
    Documentation for alarm state transition flow

Addressed by: https://review.openstack.org/447060
    Add alarm rule table in mysql for querying

Gerrit topic: https://review.openstack.org/#q,topic:group_silence_inhibit_rule,n,z

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.