Notification to admin for cinder service failure due to any abnormal conditions

Registered by Sheel Rana on 2016-01-22

In specific conditions if any service fails, admin gets no information about failure.

For ex, lets cinder-volume service is failed due to some failure at host end.
In such conditions, if user invokes any command, operation result will be unknown forever.

Lets, user invokes "cinder create" command.
In this case, control flow would be like:

 cinder-client or dashboard
                  |
        cinder-API
                  ||
     cinder-scheduler
                  |||
AMPQ(for ex, RabbitMQ)
                  ||||
       Cinder-Volume
                  |||||
      Driver(Lets LVM)
                 ||||||
           Backend

But as cinder-volume is in inactive state, we have broken link from APMQ to cinder-volume(cinder volume is not in working state).
So, messaging queue will never be able to send this message to cinder-volume for further processing.
This message will be passed only when cinder-volume service is started by someone.

But as there is no status update to admin/user, service state will be unreported and only be tracked if continous monitoring is done by admin.

Motivation behind this BP:

1. As explained above,status of cinder create operation will be "creating" forever, there will be no update in same.

2. There will be pilling up of a lot of messages on messaging queue as well.

3. User will be in dilemma why operation is in same state since long time.

4. No failure update to admin. If admin got such update, he/she will be able to recover failed service.
In current cinder behavior, admin gets to know about failed services only if continuous monitoring of status of operations or services is done.

This BP implementation depend upon https://blueprints.launchpad.net/cinder/+spec/summarymessage which is already approved.

For how these notification will be displayed to admin, refer html tagging kept on https://blueprints.launchpad.net/cinder/+spec/summarymessage

Blueprint information

Status:
Complete
Approver:
Sean McGinnis
Priority:
High
Drafter:
Sheel Rana
Direction:
Needs approval
Assignee:
Sheel Rana
Definition:
Obsolete
Series goal:
None
Implementation:
Deferred
Milestone target:
None
Completed by
Sean McGinnis on 2017-06-14

Related branches

Sprints

Whiteboard

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.