rapid firing from command line can go over quota

Bug #938317 reported by andrewsben
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Kevin L. Mitchell

Bug Description

Using a python script I was able to run a system command 20 times to extra large launch instances. Only a few of the instances actually started without erroring but with 3 starting successfully and 8 cores per I was able to go over my quota of 20 cores.

Revision history for this message
andrewsben (andrewsben) wrote :
Revision history for this message
andrewsben (andrewsben) wrote :
Revision history for this message
andrewsben (andrewsben) wrote :

script was to the effect of

for x in range(20):
    os.system('nova boot --flavor 5 --image adsfsadfdsa-fdsafdsf-adasfdsa test_%d &' %x
    x += 1

Thierry Carrez (ttx)
Changed in nova:
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Kevin L. Mitchell (klmitch) wrote :

I think fixing this may require some sort of rearchitecting of how nova does quota management. The idea I have in mind specifically is that the quota check would be modified to also store a "reservation" if quota succeeds. Once the instance (or whatever) is used, the reservation could be ticked over to the in-use count, and if it fails, it can simply be deleted. (Of course, there should also be a synchronization to remove orphaned reservations…) Quota checks would then count not just the existing {instances,cores,ram,etc.} but also the existing reservations; this would be sufficient to prevent oversubscribing quotas this way.

(Of course, there would have to be atomicity constraints on the quota check/reservation create.)

Revision history for this message
Kevin L. Mitchell (klmitch) wrote :

Here are some of my thoughts on how this could be addressed:

http://etherpad.openstack.org/quota-atomicity

Changed in nova:
assignee: nobody → Kevin L. Mitchell (klmitch)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/6774

Changed in nova:
status: Confirmed → In Progress
Changed in nova:
assignee: Kevin L. Mitchell (klmitch) → Mohammed Naser (mnaser)
Changed in nova:
assignee: Mohammed Naser (mnaser) → Kevin L. Mitchell (klmitch)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/6774
Committed: http://github.com/openstack/nova/commit/406ff304bb09f144a59448e0e9d2d01160c7d553
Submitter: Jenkins
Branch: master

commit 406ff304bb09f144a59448e0e9d2d01160c7d553
Author: Kevin L. Mitchell <email address hidden>
Date: Fri May 4 19:27:43 2012 -0500

    Rearchitect quota checking to partially fix bug 938317.

    This is a rearchitecting/rewriting of quota handling to correct the
    quota atomicity issues highlighted by bug 938317. Partially implements
    blueprint quota-refactor as well.

    This change is fairly substantial. To make it easier to review, it has been
    broken up into 3 parts. This is the first part.

    Change-Id: I805f5750c08de17487e59fe33fad0bed203188a6

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
Kevin L. Mitchell (klmitch) wrote :

Patch committed builds the infrastructure; a second patch actually uses the infrastructure and fixes this bug.

Changed in nova:
status: Fix Committed → In Progress
Revision history for this message
Mohammed Naser (mnaser) wrote :

I believe this should be closed as the new quota system is in and committed :)

Revision history for this message
Kevin L. Mitchell (klmitch) wrote :

Yep. There's still work to do, but this particular bug should be fixed.

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → folsom-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: folsom-2 → 2012.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.