Decoupled source & sink configuration, with resource discovery

The original pipeline.yaml configuration format involved an inappropriate conflation of sources (i.e. pollsters & listeners) with sinks (i.e. transformers & publishers) in a single pipeline structure.

This layering violation has blocked the introduction of support for pluggable resource discovery, which is needed for a workable SNMP-based host monitoring scheme.

We propose to replace the old pipelines with clearly separated sources and sinks, where:

* each source encapsulates meter name matching, polling interval determination, optional resource enumeration or discovery, and mapping to one or more sinks for publication

* each sink config is concerned only with the transformation rules and publication conduits

Thus samples emitted by each source are mapped to one or more sinks, so that the pipeline is no longer polluted with source-specific configuration (such as resources) that only applied to a subset of the pollsters matching the pipeline.

To avoid breaking deployements with an old-style pipeline.yaml, automatic detection of the old format will be supported, in which case a "compatibility mode" will be used.

The new format will contain two top-level elements, 'sources' and 'sinks', for example:

  ---
  sources:
     - name: host_cpu_source
       interval: 60
       meters:
             - "cpu.util.*min"
       resources:
             - "snmp://ip1"
             - "snmp://ip2"
             - "snmp://ip3"
       sinks:
             - "meter_sink"
     - name: host_memory_source
       interval: 120
       meters:
             - "memory.used"
             - "memory.total"
       resource-discovery:
             - "hosts_file"
       sinks:
             - "meter_sink"
     - name: instance_cpu_source
       interval: 60
       resource-discovery:
             - "local_instances"
       meters:
             - "cpu"
       sinks:
             - "cpu_sink"
     - name: disk_source
       interval: 600
       meters:
             - "disk.read.bytes"
             - "disk.read.requests"
             - "disk.write.bytes"
             - "disk.write.requests"
       sinks:
             - "disk_sink"
     - name: meter_source
       interval: 600
       meters:
             - "*"
       sinks:
             - "meter_sink"
  sinks:
     - name: meter_sink
       transformers:
       publishers:
             - rpc://
     - name: cpu_sink
       transformers:
             - name: "rate_of_change"
               parameters:
                   target:
                       name: "cpu_util"
                       unit: "%"
                       type: "gauge"
                       scale: "100.0 / (10**9 * (resource_metadata.cpu_number or 1))"
       publishers:
             - udp://addr
     - name: disk_sink
       transformers:
             - name: "rate_of_change"
               parameters:
                   source:
                       map_from:
                           name: "disk\\.(read|write)\\.(bytes|requests)"
                           unit: "(B|request)"
                   target:
                       map_to:
                           name: "disk.\\1.\\2.rate"
                           unit: "\\1/s"
                       type: "gauge"
       publishers:
             - rpc://

One the source/sink config is parsed, each agent would load all pollster entry_points from its stevedore ceilometer.poll.<agent> namespace, and match each pollster to a subset of the configured sources.

Polling tasks would then be set up for each unique interval with at least one (pollster object, matching source config) pair.

The agent manager (central or compute) also would retrieve the resource-discovery extensions (if any) and then each polling task would follow the sequence:

  1. assemble target list from:
     (a) resources explicitly listed in the source config (if any)
     (b) resources provided by the resource discovery extensions (if any, with appropriate caching between tasks with the same resource-discovery)

  2. call out to each associated pollster with zero or more retrieved resources

For example the compute agent's current inline logic to retreive all non-error instances running on the local host could be recast as the stevedore extension:

  [ceilometer.discover.compute]
  local_instances = ceilometer.compute.discovery:InstanceDiscoverer

whereas hosts to monitor via SMNP could be either explicitly listed in the sink config, or loaded via another file listing IPs:

  [ceilometer.discover.central]
  host_file = ceilometer.common.discovery:FileDiscoverer