Add I/O Semaphore to limit number of concurrent disk operations

Registered by Jack Ding on 2018-09-27

On a compute node there is a limited amount of disk performance, and if we load it down heavily enough it's possible to cause control-plane operations to time out or to adversely affect guest local disk performance. As such, we want to introduce an I/O semaphore (with the concurrency value set by a new config option) to limit the number of concurrent high-IOPS (Inputs/Outputs per Second) disk operations. This could reduce disk contention from image operations like image download, image format conversion, snapshot extraction, etc.

The new config option can be set in nova.conf per compute host and would be virt-driver-agnostic. It is default to 0 which means no limit.

Blueprint information

Status:
Complete
Approver:
Matt Riedemann
Priority:
Low
Drafter:
Jack Ding
Direction:
Approved
Assignee:
Jack Ding
Definition:
Approved
Series goal:
Accepted for stein
Implementation:
Implemented
Milestone target:
milestone icon stein-2
Started by
Matt Riedemann on 2018-10-31
Completed by
Matt Riedemann on 2018-11-30

Related branches

Sprints

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/io-semaphore-for-concurrent-disk-ops,n,z

Addressed by: https://review.openstack.org/609180
    Add I/O Semaphore to limit concurrent disk ops

Notes from the Stein PTG discussion about this:

https://etherpad.openstack.org/p/nova-ptg-stein

I/O semaphore
- new config option to specify the number of concurrent high-IOPS operations like image download, image conversion, snapshot extraction
- also offloaded some of these to threads
- Set in nova.conf per compute host. Code on this is very simple.
- Config option would not be driver-specific.
- See: https://github.com/starlingx-staging/stx-nova/blob/f3662904da8626634bc4e378dcecde4564eb2751/nova/compute/manager.py#L1217
- We're doing this already for live migration
- Default to 0 which means, no limit
- AGREE: this sounds like an improvement. may not be totally robust but it's better than what we have and a good first step.

-- mriedem 20181030

This blueprint was mentioned in the Nov 1 nova meeting and there were no objections to approve this specless blueprint so I'm approving for Stein. -- mriedem 20181101

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.