Improve error handling for provisioning operations
Currently we have two problems connected with error handling in provisioning part:
1. The code incorrectly handles situations when cluster was deleted by user during provisioning. In that case an arbitrary error might be raised in many places.
2. The code performs rollback only in certain places, while it could be done for any provisioning/
The following CR:
https:/
mostly fixes issue #1, but it is full of duplicate code.
The following solution is proposed instead which requires architectural changes, but rather reliably fixes both problems:
1. For both cluster creation and scaling move error handling logic to the very top: functions inside ops.py file. Once exception is caught properly process it:
a. if cluster object does not exists in DB, that means that user deleted the cluster during provisioning; handle it and return
b. if cluster object exists, log it and perform rollback
2. Do not do any checks if cluster exists outside of ops.py, except places where processing might hang indefinitely without the check.
We can employ the following rollback strategy:
For cluster creation: if anything went wrong, kill all VMs and move cluster to the Error state.
For cluster scaling: that will be long. Cluster scaling has the following stages:
1. decommission unneeded nodes (by plugin)
2. terminate unneeded nodes and create a new ones if needed (by engine). Note that both scaling up and down could be run simultaneously but in different node groups.
3. Configure and start nodes (by plugin)
My suggestion what to do if an exception occurred in the respective stage:
1. move cluster to Error state
2. kill unneeded nodes (finish scale down). Also kill new nodes, if they were created for scale up.
3. move cluster to Error state
In cases #1 and #3 it is dangerous to delete not decommissioned or not configured nodes as this can lead to data loss.
Blueprint information
- Status:
- Complete
- Approver:
- Sergey Lukjanov
- Priority:
- Medium
- Drafter:
- Dmitry Mescheryakov
- Direction:
- Approved
- Assignee:
- Andrew Lazarev
- Definition:
- Approved
- Series goal:
- Accepted for juno
- Implementation:
- Implemented
- Milestone target:
- 2014.2
- Started by
- Sergey Lukjanov
- Completed by
- Sergey Lukjanov
Related branches
Related bugs
Sprints
Whiteboard
Gerrit topic: https:/
Addressed by: https:/
Improved error handling for provisioning operations
Addressed by: https:/
Added spec for provisioning error handling
Addressed by: https:/
Fixed cluster rollback on scaling with heat engine