[HBase] Improve performance of get_resources
Now we use Meter table during get_resources. Resource table cannot be used because we have no information about timestamps of meters. So there are several variants how it can be changed:
1. (Original blueprint proposal) To avoid querying both meter and resource tables, adding timestamp to HBase resource table as part of the rowkey will keep the query to a single table. This improves the efficiency when querying resources given time period.
2. Add a separate column family to Resource table that contains timestamp info. Pros: easy for testing, Cons: may be not efficiant (to be discussed)
3. Use versions info for timestamps. Keep infinite amount of versions in meter column family. So all meter "updates" will be stored as separate version. Pros: Looks very essential for HBase Cons: Very hard for testing because versions are handled inside HBase and cannot be mocked or determined before test
Chosen variant:
Resource table contains all samles. Each resource has it's own row and this row contains all samples for this resource with prefix rts_source. It allows to query by timestamps very quickly because all entries are sorted by timestamps.
Besides, resource`s metadata is updated when new sample comes. But an oldest sample may come faster then a newest one. That's why timestamp is set manually during put in Resource table - the newest values will be automatically popped up.
Blueprint information
- Status:
- Complete
- Approver:
- Julien Danjou
- Priority:
- High
- Drafter:
- Shengjie Min
- Direction:
- Approved
- Assignee:
- Nadya Privalova
- Definition:
- Approved
- Series goal:
- Accepted for juno
- Implementation:
- Implemented
- Milestone target:
- 2014.2
- Started by
- Nadya Privalova
- Completed by
- Nadya Privalova
Related branches
Related bugs
Sprints
Whiteboard
Gerrit topic: https:/
Addressed by: https:/
[HBase] get_resource optimization
Addressed by: https:/
Make entities (Resource, User, Project) able to store lists
Work Items
Dependency tree
* Blueprints in grey have been implemented.