SQL backend to handle 'big data'

Registered by gordon chung on 2014-03-13

the collector and notification worker only works on a single thread... enabling collector to use multiple workers causes massive amount of deadlocks because of updating done to source/project/user/resource tables. these tables do nothing to help queries and only act as a bottleneck or point of deadlock when recording metering data. we should store raw Samples and not update anything... queries can be done against Sample with the same/better performance as current design.

Blueprint information

Status:
Complete
Approver:
gordon chung
Priority:
High
Drafter:
gordon chung
Direction:
Approved
Assignee:
gordon chung
Definition:
Approved
Series goal:
Accepted for juno
Implementation:
Implemented
Milestone target:
milestone icon 2014.2
Started by
gordon chung on 2014-03-14
Completed by
gordon chung on 2014-06-05

Related branches

Sprints

Whiteboard

Gerrit topic: https://review.openstack.org/#q,topic:bp/big-data-sql,n,z

Addressed by: https://review.openstack.org/80343
    improve performance of resource-list in sql
- merged

Addressed by: https://review.openstack.org/80461
    DBDeadlock exception in sql backend

Using SQL and big-data in same sentence seems funny to me :). I guess we should discuss this topic during summit and create a detailed plan of how to proceed with big-data.
alexei-kornienko 14.03.14

i should put big-data in quotes. -- gordc

Agreed that a summit discussion around this would be good...tweaks to the queries will certainly help, but the poor performance may need to be addressed by re-working the schema.
nealph 18.03.14

random thought - the queries seem to be ripe with opportunity to do concurrent queries (get a list of meter_id/resource_id and in parallel query required details for each one)... seems like a (simple) way to 'divide and conquer'
-- gordc 18.03.14

Addressed by: https://review.openstack.org/79962
    spawn multiple workers in services

Addressed by: https://review.openstack.org/93829
    Remove unused db code due to api v1 drop

Addressed by: https://review.openstack.org/94483
    refactor sql backend to improve write speed

NOTE:
write speed has been brought up to par as of https://review.openstack.org/94483... further improvements will be addressed in subsequent bps -- gordc (05.06.14)

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.