One Meter per Pollster Plugin

Registered by Doug Hellmann

We currently allow pollster plugins to return multiple counters. This makes it difficult to control which plugins are enabled and which are disabled, and led to an O(n^2) configuration loop in We should tighten the pollster plugin API up so that each instance only generates counters for one type of meter. That will make implementing them simpler, and make configuring the pipelines easier.

The argument against this approach in the past was that some plugins will ask for the same data over and over, which is inefficient. To address that, we should provide a context or cache object to the pollsters so they can save data. That context would be refreshed at the start of each invocation loop, and any pollster could add data and reuse data created from previous pollsters. This would allow the plugins to do things like scan the instance data once, and save it for other pollsters to reuse.

For backwards compatibility, we will need to support the existing plugin API (and its setup code). To do that we can change the namespace used to load the new pollsters, so we can tell the old from the new.

Blueprint information

Doug Hellmann
Doug Hellmann
Series goal:
Accepted for havana
Milestone target:
milestone icon 2013.2
Started by
Julien Danjou
Completed by
Julien Danjou

Related branches



>>>I prefix my reply with '>>>' so that it's easier to distinguish. --jyh

dhellman, I'm not sure the exact meaning of "each instance only generates counters for one type of meter". When talking about "one type of meter", you mean only counter type will return, i.e. the pollster's len(get_counter_names()) == 1? Or you mean the len(get_counter_names()) can >1, but the pipeline configuration file mush keep these counter as an entity, i.e. either request all counters from the pollster, or no counter from the pollster. --jyh

I mean that a plugin may return >1 counter, but all of them will have the same "type" value (so get_counters() returns a list but get_counter_names() would turn into get_counter_name() and just return a single string). Whether or not the list of counters returned by get_counters() is treated as a single entity or not is up to the caller, the plugin shouldn't care. -- drh

if you mean ==1 situation, I'm not sure if we really want to put such limitation to the pollster. One possibly a bit extreme example is, the pollster need query to a back end service. For each query, the backend service need setup the environment for the request and the environment setup cost is very high.So the cost of 5 times query with one counter each query is possibly ~4x higher than 1 times query with 5 counter each query. --jyh

The cache/context is meant to address that situation. The first pollster to be invoked would see, for example, that there is no information about images in the context so it fetch the metadata and store it in the context. The other pollsters that want that data would see that it is already there, and could skip retrieving it again. -- drh

>>> Such cache/context mechanism works only if the back end service provide such mechanism so that the client can cache the content needed. However, sometimes the pollster need to flush all the information each time. E.g. in the example I listed above, possibly the client benefit nothing from such cache mechanism if the client has to ask the back end service each time, and the performance different is very big for single counter query and batch query. --jyh

>>>> Do you have an actual example of that case, or is it hypothetical? -- drh

If you mean >1 situation, I'm not sure if we really want to have such constraint to the pipeline configuration file, especially if there is no checking tools. --jyh

I don't think this would change the constraints of the configuration file at all. Can you clarify what you mean there?

What in my mind is:
a) A polling task is setup as (pipelines, (pollsters, counters)). The pipelines are pipelines with same interval. The pollsters is all pollsters will be invoked required by the pipelines.For each pollster, the polling task keep the counters that need be requested. For example, pollster A can get counter a1, a2, a3, however, the pipelines in the polling task only want counter a1, a2. --jyh

This proposal simplifies that to (pipelines, pollsters). -- drh

b) When the polling task running, it call pollster's get_counters(manager, counters), where the counters is the counter type that should be returned. --jyh

That requires the pollster implementation to dispatch or filter based on the counter name, and know that sometimes it should return different types of data. I want to keep the plugins very very very simple, and leave that filtering and dispatching in a single implementation inside the code that calls the plugins. -- drh

>>> IMHO It can still be very simple depends on the pollster requirement. For pollster that that always pass back all counters, a subclass with a default function to do the filter, while the pollster implementation only need return all counters as current code and still very simple. But someone may want to implement complex pollster to achieve performance. We don't want to close the door.

c) The pollster will return counters required. Depending on implementation, it may query all counters (a1,a2, a3) but only return required one(a1, a2), or it only query (a1, a2). --jyh

Any idea?


Gerrit topic:,topic:bp/one-meter-per-plugin,n,z

Addressed by:
    Add pollster data cache

Addressed by:
    Update glance pollster to use cache

Addressed by:
    Update floating-ip pollster to use cache

Addressed by:
    Update kwapi pollster to use cache

Addressed by:
    Update swift pollster to use cache

Addressed by:
    Update compute disk I/O pollster to use cache

Addressed by:
    Update compute CPU pollster to use cache

Addressed by:
    Update compute vnic pollster to use cache

Addressed by:
    Break up compute pollsters

Addressed by:
    Separate power and energy pollsters

Addressed by:
    Make visual coding style consistent

Addressed by:
    Split up the glance pollsters

Addressed by:
    Break up the swift pollsters

Addressed by:
    Make sure plugins are named after their meters

Addressed by:
    Remove get_counter_names from the pollster plugins


Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.