Incorpate driver supplied filtering and weighing functions to be part of placement decision making

Registered by Huang Zhiteng on 2013-11-21

We have been using filter scheduler since Grizzly. While the default filters and weigher certainly do the job and help scheduler make good decision and rule out unqualified back-ends, there are also some situation when some filters/weighers become useless. For example, due to thin-provisioning or other advanced data manipulating mechanisms back-ends may have, the 'free capacity' becomes less useful we thought it would be.

What's more, different backends have different metrics that are actually useful for them, and only the manufacturer really knows what these are, which makes it hard or almost impossible to create a global filter for these kind of metrics.

For example, a foomatic super array might have an absolute maximum capacity of 1000 volumes, and might get some performance degredation if more than 75% of total space is actually used. Also it has a max volume size of 500 gig. This sort of detail is likely to be highly variable between vendors, and even between different arrays of from the same vendor.

During HongKong design summit sessions, some idea came up from RedHat engineer. He mentioned that Cinder could allow each backend to return two new functions when reporting capabilities/status, and those will be used for filtering and weighing.

    FILTER FUNCTION and GOODNESS FUNCTION, together with any values needed by these functions.
    FILTER FUNCTION is a function that evaluates to True or False, and must evaluate to True for a specific volume in order for that backend to be considered a candidate for this volume.
    GOODNESS FUNCTION is a function that evaluates to a value from one to one hundred, indicating how much of a good fit this backend thinks it is for a specific volume. 100 means 'I'm ideal' 50 means 'if nobody better is about' and 10 'really only as a last resort'. (Better guidance on these values can be provided)
    In both cases, the function has access to details of the volume, the type, and the last reported stats from the array, e.g. volume.size type.min_iops stats.capacity
To give a concrete example for our foomatic super array, it might return:
    {
        "Backend name": "Foomatic super-array 2000";
        "Filter funtion": "(stats.n_vols < 1000) AND (stats.free_space <= volume.size) AND (volume.size <= 500)";
        "Goodness function": "(stats.percent_capacity_used < 75 ? 50 + (stats.percent_capacity_used / 2) : (stats.percent_capacity_used / 2))";
        "percent_capacity_used": 25;
        "free_space": 2575;
        "n_vols": 217;
        ...
    }

The exact operations supported by the function can be decided by the implementer, but a good start would be <, >, =, <=, >=, min(X,Y,...), max(X,Y,...), +, -, *, /, (X?Y:Z), abs()

With that, scheduler is able to share some internal wisdom of each back-ends to deal with vendor specific metrics or common metrics that vendors tend to interpret differently. Meanwhile, scheduler also have filters and weighers that does global filtering/weighing. So schedulers takes two set of filters and weighers to make optimal placement decisions.

The workflow will be:
 1) back-end drivers periodically reports status and functions to scheduler;
 2) scheduler updates its cache for status and functions;
 3) when new request comes, scheduler filters back-ends using filter-chain like this:

  INPUT (back-end status) -> [BACK-END_SPECIFIC_FILTER, Global_Filters(availability_zone_filter, retry_filter, json_filter, etc)]
                                                     |
                                                    v
            [BACK-END_SPECIFIC_WEIGHTER, Global_Weigher] -> Normalized Weights for candidates -> Winner takes the job

Blueprint information

Status:
Complete
Approver:
John Griffith
Priority:
Medium
Drafter:
Huang Zhiteng
Direction:
Approved
Assignee:
Kurt Martin
Definition:
Review
Series goal:
Accepted for kilo
Implementation:
Implemented
Milestone target:
milestone icon 2015.1.0
Started by
Duncan Thomas on 2013-12-11
Completed by
Kurt Martin on 2015-01-07

Related branches

Sprints

Whiteboard

Dropping out of I3 since I can't get the damn thing to work fully

<KM> We are doing some final testing and will have a patch up for review before Paris

Gerrit topic: https://review.openstack.org/#q,topic:specs/filtering-weighing-with-driver-supplied-functions,n,z

Addressed by: https://review.openstack.org/129330
    Add driver filter for scheduler that utilizes an evaluator

Gerrit topic: https://review.openstack.org/#q,topic:bp/filtering-weighing-with-driver-supplied-functions,n,z

Addressed by: https://review.openstack.org/129987
    Add driver filter and evaluator for scheduler

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.