Improve error handling for provisioning operations

Registered by Dmitry Mescheryakov

Currently we have two problems connected with error handling in provisioning part:
1. The code incorrectly handles situations when cluster was deleted by user during provisioning. In that case an arbitrary error might be raised in many places.
2. The code performs rollback only in certain places, while it could be done for any provisioning/scaling phase.

The following CR:
https://review.openstack.org/#/c/98556
mostly fixes issue #1, but it is full of duplicate code.

The following solution is proposed instead which requires architectural changes, but rather reliably fixes both problems:

1. For both cluster creation and scaling move error handling logic to the very top: functions inside ops.py file. Once exception is caught properly process it:
  a. if cluster object does not exists in DB, that means that user deleted the cluster during provisioning; handle it and return
  b. if cluster object exists, log it and perform rollback
2. Do not do any checks if cluster exists outside of ops.py, except places where processing might hang indefinitely without the check.

We can employ the following rollback strategy:

For cluster creation: if anything went wrong, kill all VMs and move cluster to the Error state.

For cluster scaling: that will be long. Cluster scaling has the following stages:
1. decommission unneeded nodes (by plugin)
2. terminate unneeded nodes and create a new ones if needed (by engine). Note that both scaling up and down could be run simultaneously but in different node groups.
3. Configure and start nodes (by plugin)

My suggestion what to do if an exception occurred in the respective stage:
1. move cluster to Error state
2. kill unneeded nodes (finish scale down). Also kill new nodes, if they were created for scale up.
3. move cluster to Error state

In cases #1 and #3 it is dangerous to delete not decommissioned or not configured nodes as this can lead to data loss.

Blueprint information

Status:
Complete
Approver:
Sergey Lukjanov
Priority:
Medium
Drafter:
Dmitry Mescheryakov
Direction:
Approved
Assignee:
Andrew Lazarev
Definition:
Approved
Series goal:
Accepted for juno
Implementation:
Implemented
Milestone target:
milestone icon 2014.2
Started by
Sergey Lukjanov
Completed by
Sergey Lukjanov

Related branches

Sprints

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/error-handling-in-provisioning,n,z

Addressed by: https://review.openstack.org/104395
    Improved error handling for provisioning operations

Addressed by: https://review.openstack.org/104700
    Added spec for provisioning error handling

Addressed by: https://review.openstack.org/105321
    Fixed cluster rollback on scaling with heat engine

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.