EDP-centric metering

To Sahara, EDP provides not only a method for using data process services without low-level knowledge, but also the potential capability of user behavior and system status awareness for metering (and furthurmore other advanced purposes).

Assume there is no EDP. Now a user can just log in the job tracker of virtualized Hadoop cluster provisioned by Sahara, and run jobs directly, just like he/she is using a normal, physical cluster. In such a case, Sahara is in fact bypassed during the whole data processing, and can hardly collect any useful user and job information.

On the contrary, if users access services via EDP, the job description info can easily be collected and recorded by EDP. In this case, EDP can either send such raw info directly to Ceilometer, or apply local analysis for management purposes, e.g. leveraging such info EDP can meter the workload of each cluster, or the processed data size from a given user during a period of time.

This BP aims at extending EDP to collect user-input job description info. The collected info can be either feed into Ceilometer, or stored in a local DB. A plugin might be introduced to pre-process the collected info before it is sent out or stored.

This BP can be considered as a supplement of BP https://blueprints.launchpad.net/sahara/+spec/ceilometer-integration