One Meter per Pollster Plugin
We currently allow pollster plugins to return multiple counters. This makes it difficult to control which plugins are enabled and which are disabled, and led to an O(n^2) configuration loop in https:/
The argument against this approach in the past was that some plugins will ask for the same data over and over, which is inefficient. To address that, we should provide a context or cache object to the pollsters so they can save data. That context would be refreshed at the start of each invocation loop, and any pollster could add data and reuse data created from previous pollsters. This would allow the plugins to do things like scan the instance data once, and save it for other pollsters to reuse.
For backwards compatibility, we will need to support the existing plugin API (and its setup code). To do that we can change the namespace used to load the new pollsters, so we can tell the old from the new.
Blueprint information
- Status:
- Complete
- Approver:
- None
- Priority:
- High
- Drafter:
- Doug Hellmann
- Direction:
- Approved
- Assignee:
- Doug Hellmann
- Definition:
- Approved
- Series goal:
- Accepted for havana
- Implementation:
-
Implemented
- Milestone target:
-
2013.2
- Started by
- Julien Danjou
- Completed by
- Julien Danjou
Related branches
Related bugs
Sprints
Whiteboard
>>>I prefix my reply with '>>>' so that it's easier to distinguish. --jyh
dhellman, I'm not sure the exact meaning of "each instance only generates counters for one type of meter". When talking about "one type of meter", you mean only counter type will return, i.e. the pollster's len(get_
I mean that a plugin may return >1 counter, but all of them will have the same "type" value (so get_counters() returns a list but get_counter_names() would turn into get_counter_name() and just return a single string). Whether or not the list of counters returned by get_counters() is treated as a single entity or not is up to the caller, the plugin shouldn't care. -- drh
if you mean ==1 situation, I'm not sure if we really want to put such limitation to the pollster. One possibly a bit extreme example is, the pollster need query to a back end service. For each query, the backend service need setup the environment for the request and the environment setup cost is very high.So the cost of 5 times query with one counter each query is possibly ~4x higher than 1 times query with 5 counter each query. --jyh
The cache/context is meant to address that situation. The first pollster to be invoked would see, for example, that there is no information about images in the context so it fetch the metadata and store it in the context. The other pollsters that want that data would see that it is already there, and could skip retrieving it again. -- drh
>>> Such cache/context mechanism works only if the back end service provide such mechanism so that the client can cache the content needed. However, sometimes the pollster need to flush all the information each time. E.g. in the example I listed above, possibly the client benefit nothing from such cache mechanism if the client has to ask the back end service each time, and the performance different is very big for single counter query and batch query. --jyh
>>>> Do you have an actual example of that case, or is it hypothetical? -- drh
If you mean >1 situation, I'm not sure if we really want to have such constraint to the pipeline configuration file, especially if there is no checking tools. --jyh
I don't think this would change the constraints of the configuration file at all. Can you clarify what you mean there?
What in my mind is:
a) A polling task is setup as (pipelines, (pollsters, counters)). The pipelines are pipelines with same interval. The pollsters is all pollsters will be invoked required by the pipelines.For each pollster, the polling task keep the counters that need be requested. For example, pollster A can get counter a1, a2, a3, however, the pipelines in the polling task only want counter a1, a2. --jyh
This proposal simplifies that to (pipelines, pollsters). -- drh
b) When the polling task running, it call pollster's get_counters(
That requires the pollster implementation to dispatch or filter based on the counter name, and know that sometimes it should return different types of data. I want to keep the plugins very very very simple, and leave that filtering and dispatching in a single implementation inside the code that calls the plugins. -- drh
>>> IMHO It can still be very simple depends on the pollster requirement. For pollster that that always pass back all counters, a subclass with a default function to do the filter, while the pollster implementation only need return all counters as current code and still very simple. But someone may want to implement complex pollster to achieve performance. We don't want to close the door.
c) The pollster will return counters required. Depending on implementation, it may query all counters (a1,a2, a3) but only return required one(a1, a2), or it only query (a1, a2). --jyh
Any idea?
--jyh
Gerrit topic: https:/
Addressed by: https:/
Add pollster data cache
Addressed by: https:/
Update glance pollster to use cache
Addressed by: https:/
Update floating-ip pollster to use cache
Addressed by: https:/
Update kwapi pollster to use cache
Addressed by: https:/
Update swift pollster to use cache
Addressed by: https:/
Update compute disk I/O pollster to use cache
Addressed by: https:/
Update compute CPU pollster to use cache
Addressed by: https:/
Update compute vnic pollster to use cache
Addressed by: https:/
Break up compute pollsters
Addressed by: https:/
Separate power and energy pollsters
Addressed by: https:/
Make visual coding style consistent
Addressed by: https:/
Split up the glance pollsters
Addressed by: https:/
Break up the swift pollsters
Addressed by: https:/
Make sure plugins are named after their meters
Addressed by: https:/
Remove get_counter_names from the pollster plugins
Work Items
Dependency tree
* Blueprints in grey have been implemented.
