Support storing the entire event body.

Registered by Andrew Melton on 2013-07-19

Reference Branch: https://github.com/ramielrowe/ceilometer/tree/event_body

In our experiences with Stacktach, we've found it extremely valuable to store the entire event body so that new datapoints and metrics can be rolled up after the fact. This feature would allow the storing of the entire event body in Ceilometer's database. In Stacktach, event bodies are stored as single JSON strings. Due to that it makes building new metrics a very slow and painful process of selecting large subsets of events, parsing the json to a dictionary and the grabbing the data needed. A better solution would allow the indexed searching of these bodies without too much complexity.

This feature would introduce a new api model called EventBody containing two fields, the event, and the event's body as a dictionary. The reference branch above outlines the beginnings of a SQLAlchemy implementation. In SQLAlchemy, EventBodies are made up of EventBodyEntries. Each entry has four fields, the event, a key (UniqueName), index (Text), and value (Text).

Sample Dictionary with it's corresponding EventBodyEntries:
https://gist.github.com/ramielrowe/6041427

With that model it would be relatively simple to search for and grab all events where 'payload.image_meta.os_type' is 'windows' or 'linux' and add an os_type field to them. Due to the index field, no data is lost about the structure of the body and the dictionary can be rebuilt if the full body is required.

Another thing to note is that EventBodies are not made up of Traits. Traits are values of known types for which we currently see as useful. The EventBodies are made of of values of unknown type and for which we currently don't have a use for. The importance of keeping these values around is so that when a new use is discovered, we can generate that trait for past data.

Blueprint information

Status:
Complete
Approver:
None
Priority:
Undefined
Drafter:
Andrew Melton
Direction:
Needs approval
Assignee:
None
Definition:
Obsolete
Series goal:
None
Implementation:
Unknown
Milestone target:
None
Completed by
gordon chung on 2015-06-17

Related branches

Sprints

Whiteboard

Seems to me this will add significant amount of data into the database, and not mention slow down the data insertion. Would be nice to think about using some kind of callout based on search/index engine like Lucene.

is this being worked on? i think this could be restarted and wouldn't be that big a change. (at least not for some db. --gordc (19/11/14)

i'm modifying this to store unindexed body... storing full decomposed notification will blow up everything. -- gordc(6/2/15)

i actually implemented the ability to store raw notification in Events. they will be unindexed as mentioned above https://review.openstack.org/#/c/153790/ ... if you're content we can mark this obsolete -- gordc(10/3/15)

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.