I think fixing this may require some sort of rearchitecting of how nova does quota management. The idea I have in mind specifically is that the quota check would be modified to also store a "reservation" if quota succeeds. Once the instance (or whatever) is used, the reservation could be ticked over to the in-use count, and if it fails, it can simply be deleted. (Of course, there should also be a synchronization to remove orphaned reservations…) Quota checks would then count not just the existing {instances,cores,ram,etc.} but also the existing reservations; this would be sufficient to prevent oversubscribing quotas this way.
(Of course, there would have to be atomicity constraints on the quota check/reservation create.)
I think fixing this may require some sort of rearchitecting of how nova does quota management. The idea I have in mind specifically is that the quota check would be modified to also store a "reservation" if quota succeeds. Once the instance (or whatever) is used, the reservation could be ticked over to the in-use count, and if it fails, it can simply be deleted. (Of course, there should also be a synchronization to remove orphaned reservations…) Quota checks would then count not just the existing {instances, cores,ram, etc.} but also the existing reservations; this would be sufficient to prevent oversubscribing quotas this way.
(Of course, there would have to be atomicity constraints on the quota check/reservation create.)