Update Failure Recovery

Registered by Zane Bitter on 2013-11-22

Currently, stack updates are handled in an all-or-nothing kind of way. If a failure occurs, we attempt to roll back to the previous state if rollback is enabled. If the rollback fails or is disabled, we leave the stack in its failed state, but accept the old or new template (respectively) as a true representation of the current state of the stack. (This means that we could lose track of some resources and not be able to delete them.) We also prohibit updates to the stack from this point on; once an update has failed, you can only delete the stack.

We need to incrementally update the current template as resources are added, removed or modified. This will give us a valid picture of the true state when a failure occurs, allowing us to safely run updates in the future.

Blueprint information

Status:
Complete
Approver:
Steve Baker
Priority:
High
Drafter:
None
Direction:
Approved
Assignee:
Zane Bitter
Definition:
Approved
Series goal:
Accepted for juno
Implementation:
Implemented
Milestone target:
milestone icon 2014.2
Started by
Zane Bitter on 2014-05-28
Completed by
Steve Baker on 2014-09-02

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/update-failure-recovery,n,z

Addressed by: https://review.openstack.org/100044
    Update: Keep a copy of the old template for rollback

Addressed by: https://review.openstack.org/100045
    Allow raw templates to be updated in the DB

Addressed by: https://review.openstack.org/100046
    Update template incrementally as resources change

Addressed by: https://review.openstack.org/100047
    Update: persist current template on change

Addressed by: https://review.openstack.org/96008
    Unit tests: use ResourceDefinition to test facades

Addressed by: https://review.openstack.org/96009
    Use ResourceDefinition to generate UpdatePolicy

Addressed by: https://review.openstack.org/96010
    Get deletion policy from ResourceDefinition

Addressed by: https://review.openstack.org/96011
    Get the resource type from ResourceDefinition

Addressed by: https://review.openstack.org/96012
    Get metadata from ResourceDefinition

Addressed by: https://review.openstack.org/96013
    Get resource description from ResourceDefinition

Addressed by: https://review.openstack.org/96004
    Unit tests: Always use ResourceDefinition for updates

Addressed by: https://review.openstack.org/96005
    Unit tests: always use ResourceDefinition for handle_update()

Addressed by: https://review.openstack.org/96006
    Unit tests: Name resources the same as in template

Addressed by: https://review.openstack.org/96007
    Use ResourceDefinition to generate Properties

Addressed by: https://review.openstack.org/96922
    RPC API: Really don't include metadata in resource list

Addressed by: https://review.openstack.org/96923
    Update: Make addition/removal of resources more explicit

Addressed by: https://review.openstack.org/96926
    Refactor resource initialisation from DB

Addressed by: https://review.openstack.org/96927
    Load resources using Resource.load_all_from_stack()

Addressed by: https://review.openstack.org/96924
    Implement Stack loading from DB as a separate function

Addressed by: https://review.openstack.org/96925
    List stacks using Stack.load_all()

Gerrit topic: https://review.openstack.org/#q,topic:typeless-function-plugins,n,z

Addressed by: https://review.openstack.org/112930
    Store properties data in database

Addressed by: https://review.openstack.org/112934
    Diff against stored properties during update

Addressed by: https://review.openstack.org/112938
    Allow an update after a failure

Addressed by: https://review.openstack.org/112941
    Don't stop creates/updates immediately on error

Addressed by: https://review.openstack.org/112937
    Use ResourceDefinition as 'before' in resource updates

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.