Update Failure Recovery
Currently, stack updates are handled in an all-or-nothing kind of way. If a failure occurs, we attempt to roll back to the previous state if rollback is enabled. If the rollback fails or is disabled, we leave the stack in its failed state, but accept the old or new template (respectively) as a true representation of the current state of the stack. (This means that we could lose track of some resources and not be able to delete them.) We also prohibit updates to the stack from this point on; once an update has failed, you can only delete the stack.
We need to incrementally update the current template as resources are added, removed or modified. This will give us a valid picture of the true state when a failure occurs, allowing us to safely run updates in the future.
Blueprint information
- Status:
- Complete
- Approver:
- Steve Baker
- Priority:
- High
- Drafter:
- None
- Direction:
- Approved
- Assignee:
- Zane Bitter
- Definition:
- Approved
- Series goal:
- Accepted for juno
- Implementation:
-
Implemented
- Milestone target:
-
2014.2
- Started by
- Zane Bitter
- Completed by
- Steve Baker
Related branches
Related bugs
Sprints
Whiteboard
Gerrit topic: https:/
Addressed by: https:/
Update: Keep a copy of the old template for rollback
Addressed by: https:/
Allow raw templates to be updated in the DB
Addressed by: https:/
Update template incrementally as resources change
Addressed by: https:/
Update: persist current template on change
Addressed by: https:/
Unit tests: use ResourceDefinition to test facades
Addressed by: https:/
Use ResourceDefinition to generate UpdatePolicy
Addressed by: https:/
Get deletion policy from ResourceDefinition
Addressed by: https:/
Get the resource type from ResourceDefinition
Addressed by: https:/
Get metadata from ResourceDefinition
Addressed by: https:/
Get resource description from ResourceDefinition
Addressed by: https:/
Unit tests: Always use ResourceDefinition for updates
Addressed by: https:/
Unit tests: always use ResourceDefinition for handle_update()
Addressed by: https:/
Unit tests: Name resources the same as in template
Addressed by: https:/
Use ResourceDefinition to generate Properties
Addressed by: https:/
RPC API: Really don't include metadata in resource list
Addressed by: https:/
Update: Make addition/removal of resources more explicit
Addressed by: https:/
Refactor resource initialisation from DB
Addressed by: https:/
Load resources using Resource.
Addressed by: https:/
Implement Stack loading from DB as a separate function
Addressed by: https:/
List stacks using Stack.load_all()
Gerrit topic: https:/
Addressed by: https:/
Store properties data in database
Addressed by: https:/
Diff against stored properties during update
Addressed by: https:/
Allow an update after a failure
Addressed by: https:/
Don't stop creates/updates immediately on error
Addressed by: https:/
Use ResourceDefinition as 'before' in resource updates
Work Items
Dependency tree
* Blueprints in grey have been implemented.
