Improve error handling for provisioning operations

Registered by Dmitry Mescheryakov

Currently we have two problems connected with error handling in provisioning part:
1. The code incorrectly handles situations when cluster was deleted by user during provisioning. In that case an arbitrary error might be raised in many places.
2. The code performs rollback only in certain places, while it could be done for any provisioning/scaling phase.

The following CR:
mostly fixes issue #1, but it is full of duplicate code.

The following solution is proposed instead which requires architectural changes, but rather reliably fixes both problems:

1. For both cluster creation and scaling move error handling logic to the very top: functions inside file. Once exception is caught properly process it:
  a. if cluster object does not exists in DB, that means that user deleted the cluster during provisioning; handle it and return
  b. if cluster object exists, log it and perform rollback
2. Do not do any checks if cluster exists outside of, except places where processing might hang indefinitely without the check.

We can employ the following rollback strategy:

For cluster creation: if anything went wrong, kill all VMs and move cluster to the Error state.

For cluster scaling: that will be long. Cluster scaling has the following stages:
1. decommission unneeded nodes (by plugin)
2. terminate unneeded nodes and create a new ones if needed (by engine). Note that both scaling up and down could be run simultaneously but in different node groups.
3. Configure and start nodes (by plugin)

My suggestion what to do if an exception occurred in the respective stage:
1. move cluster to Error state
2. kill unneeded nodes (finish scale down). Also kill new nodes, if they were created for scale up.
3. move cluster to Error state

In cases #1 and #3 it is dangerous to delete not decommissioned or not configured nodes as this can lead to data loss.

Blueprint information

Sergey Lukjanov
Dmitry Mescheryakov
Andrew Lazarev
Series goal:
Accepted for juno
Milestone target:
milestone icon 2014.2
Started by
Sergey Lukjanov
Completed by
Sergey Lukjanov

Related branches



Gerrit topic:,topic:bp/error-handling-in-provisioning,n,z

Addressed by:
    Improved error handling for provisioning operations

Addressed by:
    Added spec for provisioning error handling

Addressed by:
    Fixed cluster rollback on scaling with heat engine


Work Items

This blueprint contains Public information 
Everyone can see this information.