Introduce rules API

Registered by Dmitry Tantsur

This blueprint series is the proper upstream implementation of our downstream work that was PoC-ed under name of ahc-tools (https://github.com/rdo-management/ahc-tools).

Bare metal can be very different. Simple matching that is achieved by using nova flavors is usually not enough. Fortunately, nova and ironic support "capabilities" property that allows matching on anything. The only task is populating this "capabilities" field. This spec allows doing it automatically using "rules" to be executed over data. Very primitive example of such rule is "cpu count == 2 and local_gb between 100 and 200 GiB". The same rules engine will allow, for example, to fail introspection based on some benchmark value.

We'll introduce API for defining rules that will run during data processing phase.

POST /v1/rules
GET/PUT/DELETE /v1/rules/<RULE_UUID>
Request body for POST/PUT and response body for GET is the following:
    {
        "description": "human-readable description",
        "conditions": [...],
        "actions": [...],
    }

"conditions" list defines expressions to run on introspection data. "actions" field defines actions to be taken if data matches rules.

Example:

    {
      "conditions": [
        {"field": "inventory.disk.*.size",
         "op": "gt",
         "value": 4},
        {"field": "inventory.network.*.ipv4",
         "op": "net",
         "value": "192.0.2.0/24"},
        {"field": "memory_mb",
         "op": "ge",
         "value": 4096},
      ],
      "actions": [
        {"action": "set-capability",
          "name": "much-memory",
          "value": "wow"},
        {"action": "append-attribute",
          "key": "/extra/matched",
          "value": "much-memory"}
      ]
    }

I. Conditions

Conditions API is inspired by Ceilometer query API [1]. Conditions are joined by "AND" operation. Each condition is of the following form:

    {
        "field": <FIELD_DEF>,
        "op": <OP>,
        "value": <OP ARGUMENT>,
        "multiple": <HOW TO TREAT MULTIPLE>
    }

The field is a JSON path expression as defined by [2]. Optional "multiple" argument defines what to do if an expression yield multiple results. Accepted values:
* any (the default) - expect any value to match
* all - expect all values to match
* first - expect the first value to match

The following operations will be initially supported: lt, le, eq, ne, ge, gt (comparison), net (ip address is in a given network). If reference value is integer, inspected value will also be converted to integer, the same for float. This list will be plugin-driven.

II. Actions

Each action is a dict:

    {
        "action": <ACTION-TYPE>,
        ...
    }

(see example above).

Each action is expected to be revertable, so that its effect can be unapplied if condition didn't match. See plugins section below for details.
The following actions will be initially supported:

* set-capability - set a capability on a node. Arguments: name - capability name, value - value to set. If condition didn't match, this capability will be removed.
* fail - fail introspection. Arguments: message - fail message, maintenance (optional, default to false) whether to put node in maintenance mode.
* set-attribute - set any attribute on a node. Arguments: key - attribute path, e.g. /extra/key, value - value to set, cleanup - value to set if the condition didn't match (optional, defaults to deleting attribute).
* append-attribute - the same as set-attribute, but treat attribute as a list and append value, remove value on cleanup.

III. Plugins

To simplify adding new actions or conditions, make them pluggable via entry points: ironic_inspector.rules.actions and ironic_inspector.rules.conditions.
Condition interface:
  def validate(self, params): validate arguments, default implementation requires non-empty 'value'.
  def check(self, node_info, field, params): check condition for field value.
  ALLOW_NONE attribute will define whether to pass missing field as None value. Default values of False means that condition will fail, if field is not found.

Action interface:
  def validate(self, params): validate arguments, default implementation requires no arguments.
  def apply(self, node_info, params): apply action to a node.
  def cleanup(self. node_info, params): cleanup node if conditions didn't match.

In all cases, 'params' is a dictionary with everything from an appropriate condition/action JSON representation, except for required field (like 'op', 'field', etc).

All cleanup calls are always made before any of apply calls, so e.g. it's safe to delete attribute which might be added by another rule.

[1] http://docs.openstack.org/developer/ceilometer/webapi/v2.html#filtering-queries
[2] https://pypi.python.org/pypi/jsonpath-rw

Blueprint information

Status:
Complete
Approver:
Dmitry Tantsur
Priority:
High
Drafter:
Dmitry Tantsur
Direction:
Approved
Assignee:
Dmitry Tantsur
Definition:
Approved
Series goal:
Accepted for liberty
Implementation:
Implemented
Milestone target:
milestone icon 2.2.0
Started by
Dmitry Tantsur
Completed by
Dmitry Tantsur

Related branches

Sprints

Whiteboard

yuikotakada:thank you for posting a bp. I have a question. Because it is backport of our downstream "ahc-tools", so that I've checked https://github.com/rdo-management/ahc-tools. This bp intend to API, but there seems to be no codes about API in above github. What does it mean?
dtantsur: yeah, a bit of clarification needed: ahc-tools is just a poc, it does not have API. A proper implementation should.
yuikotakada: OK, I see. I misunderstood that all we need is copy ahc-tools and paste into ironic inspector. BTW, "profile matching" feature seems not ironic inspector's work, but nova-scheduler filter's work. Is it incorrect?
dtantsur: well, not exactly (putting aside that some people want ironic + inspector without nova). The whole idea of discoverable properties does not exists in nova: virtual machines have nothing to discover, you can set everything. Putting this in nova will mean introducing one more "foreign" abstraction that is only needed for ironic.
yuikotakada: Ah, for example network information, right? :) So...I'd like to more deep dive. About API, PUT means update, right? I think we need also POST API which creates a new profile, WDYT? And I assume <PROFILE> is uuid and will be generated automatically by POST API, is it correct?
dtantsur: I was planning <PROFILE> as a human-readable name (e.g. "compute"), thus PUT API to create/update it.
dtantsur: After some discussions, I made this blueprint substantially more generic.
dtantsur: One more rethink of this spec. Pluggable actions and conditions, actions are expected to be revertable.
yuikotakada:I have 2 questions. (1)POST /v1/rules and GET/PUT/DELETE /v1/rules/<RULE_UUID> will just create/get information/update/delete rule itself, in other words, just changes Ironic Inspector's database, right? And then, by which API, node capability will be updated? (2)can rules API be used by Ironic?
yuikotakada: I refreshed this page now...It wouldn't be nice "RULE_NAME is either set or UUID is generated instead.", IMO...I prefer having both uuid and name(or description) columns
dtantsur: UUID + description sound nice indeed. Re update: there won't be API for now, it will happen during data processing.
yuikotakada: So that means, we need to fix IPA to get rules, right?
dtantsur: I doubt it, why? Our processing happens on server side (see process.py).
yuikotakada: How servers can recognize which kind of rules there are without fixing IPA? I assumed IPA should get rules from Inspector, is it incorrect? I'd like to know the workflow...could you please?
dtantsur: 3rd paragraph (maybe I should expand it): rules that will run during data processing phase. Processing happens in process.py, not in IPA (which we still don't have support). We might need to update the ramdisk to get more data to match on, but that's another story..
yuikotakada: Oh, I see. in process.py, inspection data will be compared with rule.

Gerrit topic: https://review.openstack.org/#q,topic:bp/rules,n,z

Addressed by: https://review.openstack.org/207449
    Split common database code into ironic_inspector.db

Addressed by: https://review.openstack.org/208375
    [WIP] Add introspection rules support

Client change: https://review.openstack.org/#/c/223096/

Addressed by: https://review.openstack.org/225173
    Add missing plugins for introspection rules

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.