Direct MySQLdb implementation

Registered by Chris Behrens on 2013-02-11

With the move of all DB calls for nova-compute going to nova-conductor, it becomes important to have efficient DB calls that do not block the whole python process. It's been found that:

1) sqlalchemy calls are tremendously slower than using direct calls with mysqldb. Even when using more low level sqlalchemy calls without ORM, this is true.
2) Using eventlet.tpool with sqlalchemy falls down. sqlalchemy is doing something that is exercising bugs with eventlet.tpool or is doing something incompatible. We end up in a state where the whole process is hung for reasons that are not extremely easy to track down. Even if tpool worked with sqlalchemy, the improvements by using MySQLdb directly are significant enough to provide it as an implementation option.

See: http://paste.openstack.org/show/31019/

The goal here will be to implement DB api calls 1 by 1, falling back to sqlalchemy for methods not implemented yet.

Blueprint information

Status:
Started
Approver:
Russell Bryant
Priority:
Undefined
Drafter:
Chris Behrens
Direction:
Needs approval
Assignee:
Chris Behrens
Definition:
Drafting
Series goal:
None
Implementation:
Started
Milestone target:
None
Started by
Russell Bryant on 2013-05-02

Related branches

Sprints

Whiteboard

I think that this bp depends on db-api-tests. -- Boris Pavlovic

Gerrit topic: https://review.openstack.org/#q,topic:bp/db-mysqldb-impl,n,z

Addressed by: https://review.openstack.org/53019
    MySQLdb: Add framework for mysqldb impl

Addressed by: https://review.openstack.org/53020
    MySQLdb: Add bandwidth usage update

Addressed by: https://review.openstack.org/53021
    MySQLdb: Add instance_get methods

Addressed by: https://review.openstack.org/53022
    MySQLdb: Add instance destroy

Addressed by: https://review.openstack.org/53023
    MySQLdb: Adds info_cache get/update/delete.

Addressed by: https://review.openstack.org/53024
    MySQLdb: Adds instance_update calls.

Unapproved - please re-submit via nova-spec --johnthetubagy (20th March 2014)

Removed from next, as next is now reserved for near misses from the last milestone --johnthetubaguyI think that this bp depends on db-api-tests. -- Boris Pavlovic

Gerrit topic: https://review.openstack.org/#q,topic:bp/db-mysqldb-impl,n,z

Addressed by: https://review.openstack.org/53019
    MySQLdb: Add framework for mysqldb impl

Addressed by: https://review.openstack.org/53020
    MySQLdb: Add bandwidth usage update

Addressed by: https://review.openstack.org/53021
    MySQLdb: Add instance_get methods

Addressed by: https://review.openstack.org/53022
    MySQLdb: Add instance destroy

Addressed by: https://review.openstack.org/53023
    MySQLdb: Adds info_cache get/update/delete.

Addressed by: https://review.openstack.org/53024
    MySQLdb: Adds instance_update calls.

Unapproved - please re-submit via nova-spec --johnthetubagy (20th March 2014)

Removed from next, as next is now reserved for near misses from the last milestone --johnthetubaguy

Marking this blueprint as definition: Drafting. If you are still working on this, please re-submit via nova-specs. If not, please mark as obsolete, and add a quick comment to describe why. --johnthetubaguy (20th April 2014)I think that this bp depends on db-api-tests. -- Boris Pavlovic

Gerrit topic: https://review.openstack.org/#q,topic:bp/db-mysqldb-impl,n,z

Addressed by: https://review.openstack.org/53019
    MySQLdb: Add framework for mysqldb impl

Addressed by: https://review.openstack.org/53020
    MySQLdb: Add bandwidth usage update

Addressed by: https://review.openstack.org/53021
    MySQLdb: Add instance_get methods

Addressed by: https://review.openstack.org/53022
    MySQLdb: Add instance destroy

Addressed by: https://review.openstack.org/53023
    MySQLdb: Adds info_cache get/update/delete.

Addressed by: https://review.openstack.org/53024
    MySQLdb: Adds instance_update calls.

Unapproved - please re-submit via nova-spec --johnthetubagy (20th March 2014)

Removed from next, as next is now reserved for near misses from the last milestone --johnthetubaguyI think that this bp depends on db-api-tests. -- Boris Pavlovic

Gerrit topic: https://review.openstack.org/#q,topic:bp/db-mysqldb-impl,n,z

Addressed by: https://review.openstack.org/53019
    MySQLdb: Add framework for mysqldb impl

Addressed by: https://review.openstack.org/53020
    MySQLdb: Add bandwidth usage update

Addressed by: https://review.openstack.org/53021
    MySQLdb: Add instance_get methods

Addressed by: https://review.openstack.org/53022
    MySQLdb: Add instance destroy

Addressed by: https://review.openstack.org/53023
    MySQLdb: Adds info_cache get/update/delete.

Addressed by: https://review.openstack.org/53024
    MySQLdb: Adds instance_update calls.

Unapproved - please re-submit via nova-spec --johnthetubagy (20th March 2014)

Removed from next, as next is now reserved for near misses from the last milestone --johnthetubaguy

Marking this blueprint as definition: Drafting. If you are still working on this, please re-submit via nova-specs. If not, please mark as obsolete, and add a quick comment to describe why. --johnthetubaguy (20th April 2014)

Marking this blueprint as definition: Drafting. If you are still working on this, please re-submit via nova-specs. If not, please mark as obsolete, and add a quick comment to describe why. --johnthetubaguy (20th April 2014)

I believe that #2 wrt eventlet was supposed to be addressed by using pymysql rather than mysql-python, which happened in liberty: https://review.openstack.org/#/c/184392/ (mriedem 2015-10-29)

(?)

Work Items