Handling a Down Cell
(1) Nova should be able to list instances and services even if it cannot connect to the DB of a single cell which is not the current scenario.
(2) Similarly, quota calculations for the VM creations today ignore resources in a down cell which is wrong. A plausible solution for this as discussed in the PTG is to block VM creations for those users who have VMs in the down cell and allow creations otherwise.
However for both the above scenarios to work, we would need the information regarding which instances are deleted and which are not, in the nova_api DB so that this info can be used when a cell goes down. This Spec proposes to add a new 'queued_for_delete' column to the instance_mapping table for handling nova list, nova service-list and VM creation (with correct quota) operations.
Blueprint information
- Status:
- Complete
- Approver:
- Matt Riedemann
- Priority:
- High
- Drafter:
- Surya Seetharaman
- Direction:
- Approved
- Assignee:
- Surya Seetharaman
- Definition:
- Approved
- Series goal:
- Accepted for stein
- Implementation:
-
Implemented
- Milestone target:
-
stein-3
- Started by
- Matt Riedemann
- Completed by
- Matt Riedemann
Related branches
Related bugs
| Bug #1726301: Nova should list instances even if it can't connect to a cell DB | Fix Released |
| Bug #1771810: Quota calculation connects to all available cells | Fix Released |
Sprints
Whiteboard
Gerrit topic: https:/
Addressed by: https:/
Handling a down cell
Addressed by: https:/
[WIP] Add queued for delete to instance_mappings table.
Addressed by: https:/
[WIP/POC] Add queued_for_delete field to InstanceMapping object
Addressed by: https:/
[WIP/POC] Updating queued_for_delete from instance_destroy()
Gerrit topic: https:/
Addressed by: https:/
[POC] Graceful handling of nova-list when a cell is down
Addressed by: https:/
Make nova list ignore down cells
Addressed by: https:/
[Ugly POC] Addition of "unavailable_
Approved post spec-freeze as a high priority item for Rocky. -- mriedem 20180709
Addressed by: https:/
Fix nits in the handling down cell spec
Addressed by: https:/
Online migration tool for populating queued-for-delete
Addressed by: https:/
Online data migration for queued_for_delete flag
Gerrit topic: https:/
We're past feature freeze for Rocky, so this must be deferred. Please re-propose the spec for Stein if you'd like to work on it next cycle. -- melwitt 20180727
Addressed by: https:/
Add get_by_
Addressed by: https:/
API microversion bump for handling-down-cell
Addressed by: https:/
Return a minimal construct for nova show when a cell is down
Addressed by: https:/
Return a minimal construct for nova service-list when a cell is down
Addressed by: https:/
Batch results per cell when doing cross-cell listing
Addressed by: https:/
List instances from all cells explicitly
Addressed by: https:/
Make instance_list perform per-cell batching
Addressed by: https:/
Record cell success/
Addressed by: https:/
Make CELL_TIMEOUT a constant
Addressed by: https:/
Stash the cell uuid on the context when targeting
Addressed by: https:/
Make RecordWrapper record RequestContext and expose cell_uuid
Gerrit topic: https:/
Addressed by: https:/
[WIP] Add scatter_
Addressed by: https:/
Handling a down cell
Re-approved for Stein. -- mriedem 20180824
Gerrit topic: https:/
Addressed by: https:/
Merge extended_volumes extension response into server view builder
Addressed by: https:/
Making instance/migration listing skipping down cells configurable
Addressed by: https:/
Modify get_by_
Addressed by: https:/
[WIP] Refactor scatter-gather utility to return exception objects
Addressed by: https:/
Make CellDatabases fixture reentrant
Addressed by: https:/
[WIP] Make _instances_
Addressed by: https:/
[WIP] Add os_compute_
Addressed by: https:/
Add DownCellFixture
Gerrit topic: https:/
Addressed by: https:/
Modify InstanceMapping
Addressed by: https:/
Plumbing for ignoring list_records_
Addressed by: https:/
Plumbing for allowing the all-tenants filter
Addressed by: https:/
Plumbing required in ViewBuilder to construct partial results
Addressed by: https:/
API microversion 2.68: Handles Down Cells Documentation
Addressed by: https:/
Add context.
Addressed by: https:/
[Doc Fix]Best practices for effectively tolerating down cells
Addressed by: https:/
Add testing guide for down cells
Work Items
Dependency tree
* Blueprints in grey have been implemented.
