Count quota usage from placement

Registered by melanie witt on 2017-10-03

In Pike, we re-architected the quota system to count actual resource usage
instead of using reservations and tracking quota usages in a separate database
table. We're counting resources like instances, CPU, and RAM by querying each
cell database and aggregating the results per project and per user. This
approach has a couple of downsides: it is not too efficient and it is
susceptible to undesirable behavior if a cell becomes unavailable. If a cell
becomes unavailable, resources in its database cannot be counted and will not
be included in resource usage until the cell returns. Cells could become
unavailable if an operator is performing maintenance on a cell or if a cell
database is experiencing problems and we cannot connect to it.

We can make resource usage counting for quotas much more efficient and
resilient to temporary cell outages by querying placement for resource usage
instead of reading separate cell databases.

Blueprint information

Status:
Complete
Approver:
Jay Pipes
Priority:
High
Drafter:
melanie witt
Direction:
Approved
Assignee:
melanie witt
Definition:
Approved
Series goal:
Accepted for train
Implementation:
Implemented
Milestone target:
milestone icon train-1
Started by
melanie witt on 2019-02-15
Completed by
Matt Riedemann on 2019-06-03

Related branches

Sprints

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/count-quota-usage-from-placement,n,z

Addressed by: https://review.openstack.org/509042
    Propose counting quota usage from placement and API database

Deferring this from Rocky since we never reached consensus on the spec in time for the Rocky spec freeze. -- mriedem 20180613

Addressed by: https://review.openstack.org/628702
    Add more detail and fix code block formatting

Spec merged today, approved for Stein. -- melwitt 20190109

Addressed by: https://review.openstack.org/633349
    Add user_id column to the instance_mappings table

Addressed by: https://review.openstack.org/633350
    Add user_id field to InstanceMapping

Addressed by: https://review.openstack.org/633351
    Add online data migration for populating user_id

Addressed by: https://review.openstack.org/638072
    WIP Add get_counts() to InstanceMappingList

Addressed by: https://review.openstack.org/638073
    WIP Count instances from mappings and cores/ram from placement

Addressed by: https://review.openstack.org/638324
    Use instance mappings to count server group members

Addressed by: https://review.openstack.org/638574
    Populate InstanceMapping.user_id during migrations and schedules

I'm deferring this to Train since the data model and migration changes merged in Stein but the API usage did not. -- mriedem 20190311

Re-approved for Train. -- mriedem 20190321

Gerrit topic: https://review.openstack.org/#/q/topic:bp/count-quota-usage-from-placement

Addressed by: https://review.openstack.org/653145
    Add get_usages_counts_for_quota to SchedulerReportClient

Addressed by: https://review.openstack.org/653146
    Set [quota]count_usage_from_placement = True in nova-next

Addressed by: https://review.openstack.org/653845
    Add documentation for counting quota usage from placement

Gerrit topic: https://review.opendev.org/#/q/topic:bp/count-quota-usage-from-placement

Addressed by: https://review.opendev.org/638073
    Count instances from mappings and cores/ram from placement

Addressed by: https://review.opendev.org/653146
    Set [quota]count_usage_from_placement = True in nova-next

Addressed by: https://review.opendev.org/638324
    Use instance mappings to count server group members

Addressed by: https://review.opendev.org/653845
    Add documentation for counting quota usage from placement

Addressed by: https://review.opendev.org/662028
    DNM: Test count_usage_from_placement=True with postgresql

Addressed by: https://review.opendev.org/662056
    Follow up for counting quota usage from placement

Addressed by: https://review.opendev.org/662130
    Amend count-quota-usage-from-placement to reflect implementation

The main functional part of this are merged, only remaining is a spec amendment, so I'm going to mark this complete for Train. -- mriedem 20190603

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.