Count quota usage from placement
In Pike, we re-architected the quota system to count actual resource usage
instead of using reservations and tracking quota usages in a separate database
table. We're counting resources like instances, CPU, and RAM by querying each
cell database and aggregating the results per project and per user. This
approach has a couple of downsides: it is not too efficient and it is
susceptible to undesirable behavior if a cell becomes unavailable. If a cell
becomes unavailable, resources in its database cannot be counted and will not
be included in resource usage until the cell returns. Cells could become
unavailable if an operator is performing maintenance on a cell or if a cell
database is experiencing problems and we cannot connect to it.
We can make resource usage counting for quotas much more efficient and
resilient to temporary cell outages by querying placement for resource usage
instead of reading separate cell databases.
Blueprint information
- Status:
- Complete
- Approver:
- Jay Pipes
- Priority:
- High
- Drafter:
- melanie witt
- Direction:
- Approved
- Assignee:
- melanie witt
- Definition:
- Approved
- Series goal:
- Accepted for train
- Implementation:
-
Implemented
- Milestone target:
-
train-1
- Started by
- melanie witt
- Completed by
- Matt Riedemann
Related branches
Sprints
Whiteboard
Gerrit topic: https:/
Addressed by: https:/
Propose counting quota usage from placement and API database
Deferring this from Rocky since we never reached consensus on the spec in time for the Rocky spec freeze. -- mriedem 20180613
Addressed by: https:/
Add more detail and fix code block formatting
Spec merged today, approved for Stein. -- melwitt 20190109
Addressed by: https:/
Add user_id column to the instance_mappings table
Addressed by: https:/
Add user_id field to InstanceMapping
Addressed by: https:/
Add online data migration for populating user_id
Addressed by: https:/
WIP Add get_counts() to InstanceMappingList
Addressed by: https:/
WIP Count instances from mappings and cores/ram from placement
Addressed by: https:/
Use instance mappings to count server group members
Addressed by: https:/
Populate InstanceMapping
I'm deferring this to Train since the data model and migration changes merged in Stein but the API usage did not. -- mriedem 20190311
Re-approved for Train. -- mriedem 20190321
Gerrit topic: https:/
Addressed by: https:/
Add get_usages_
Addressed by: https:/
Set [quota]
Addressed by: https:/
Add documentation for counting quota usage from placement
Gerrit topic: https:/
Addressed by: https:/
Count instances from mappings and cores/ram from placement
Addressed by: https:/
Set [quota]
Addressed by: https:/
Use instance mappings to count server group members
Addressed by: https:/
Add documentation for counting quota usage from placement
Addressed by: https:/
DNM: Test count_usage_
Addressed by: https:/
Follow up for counting quota usage from placement
Addressed by: https:/
Amend count-quota-
The main functional part of this are merged, only remaining is a spec amendment, so I'm going to mark this complete for Train. -- mriedem 20190603
Work Items
Dependency tree
* Blueprints in grey have been implemented.
