[HBase] Improve performance of get_resources

Registered by Shengjie Min on 2013-05-13

Now we use Meter table during get_resources. Resource table cannot be used because we have no information about timestamps of meters. So there are several variants how it can be changed:
1. (Original blueprint proposal) To avoid querying both meter and resource tables, adding timestamp to HBase resource table as part of the rowkey will keep the query to a single table. This improves the efficiency when querying resources given time period.
2. Add a separate column family to Resource table that contains timestamp info. Pros: easy for testing, Cons: may be not efficiant (to be discussed)
3. Use versions info for timestamps. Keep infinite amount of versions in meter column family. So all meter "updates" will be stored as separate version. Pros: Looks very essential for HBase Cons: Very hard for testing because versions are handled inside HBase and cannot be mocked or determined before test

Chosen variant:

Resource table contains all samles. Each resource has it's own row and this row contains all samples for this resource with prefix rts_source. It allows to query by timestamps very quickly because all entries are sorted by timestamps.
Besides, resource`s metadata is updated when new sample comes. But an oldest sample may come faster then a newest one. That's why timestamp is set manually during put in Resource table - the newest values will be automatically popped up.

Blueprint information

Status:
Complete
Approver:
Julien Danjou
Priority:
High
Drafter:
Shengjie Min
Direction:
Approved
Assignee:
Nadya Privalova
Definition:
Approved
Series goal:
Accepted for juno
Implementation:
Implemented
Milestone target:
milestone icon 2014.2
Started by
Nadya Privalova on 2014-04-03
Completed by
Nadya Privalova on 2014-06-17

Related branches

Sprints

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/hbase-resource-rowkey-enhancement,n,z

Addressed by: https://review.openstack.org/87249
    [HBase] get_resource optimization

Addressed by: https://review.openstack.org/78244
    Make entities (Resource, User, Project) able to store lists

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.