Schedule actions to call after database transaction is closed

Registered by Ihar Hrachyshka

In Neutron, we experience lots of db lock timeout issues like:
https://launchpad.net/bugs/1330955
https://launchpad.net/bugs/1332577

or:
https://launchpad.net/bugs/1330638
https://launchpad.net/bugs/1330639
https://launchpad.net/bugs/1330644

Those failures are triggered by eventlet green thread switched while under transaction and a row in db being locked. The thread is switched to another thread that tries to access the same (locked) row, it locks waiting for the object to be released, and this results in timeout and failure to process request.

Green thread switch does not occur randomly, it's triggered by eventlet when calling to specific (monkey patched) methods from stdlib. This includes network operations.

One common problem in Neutron is that we may be interested in sending a notification as a result of an operation. Such notification may result in green thread switch. This is fine as long as we notify from outside any active transaction. Most of the code is written with this assumption, as follows:

def example_method(self, context, ...):
    with context.session.begin(subtransactions=True):
        ... do db work ...
    ... send notification ...

This approach is fine as long as the method called from no-transaction context. It then guarantees that notifications are sent after transaction created by the method is closed.

The problem arises when the same code is called from another method that opens transaction. In that case, notifier is called from db context (even though the author of the called method didn't mean it), and then a failure may occur.

One of the approach to fix those db lock timeouts is to make notification requests explicit, as in: https://review.openstack.org/#/c/100934/

The problem with that approach is that it mixes implementation details with API, exposing notification internals to callers that should not be interested in a way that notifications are handled by the called function. While there is still a value for patching the code for most common db lock timeouts until a more general approach is implemented, we need to find another way to handle those situations long term.

This blueprint proposes the following:
- instead of calling to a notifier directly from methods, schedule them to be executed once database session is indeed closed. This means adding a new method to schedule any funcall with determined arguments in session context. If transaction is not active, the scheduled function will be called right away.
- modify context.session object's __exit__ function to execute all scheduled actions once transaction is closed indeed.
- modify all the code that currently calls to notifier directly to use new scheduling method.

Blueprint information

Status:
Complete
Approver:
None
Priority:
Undefined
Drafter:
Ihar Hrachyshka
Direction:
Needs approval
Assignee:
Ihar Hrachyshka
Definition:
Obsolete
Series goal:
None
Implementation:
Unknown
Milestone target:
None
Completed by
Ihar Hrachyshka

Related branches

Sprints

Whiteboard

Proposal is abandoned in favour of: https://blueprints.launchpad.net/neutron/+spec/switch-to-mysql-connector

Gerrit topic: https://review.openstack.org/#q,topic:bp/schedule-actions-for-db-transaction-closed,n,z

Addressed by: https://review.openstack.org/103540
    Mechanism to schedule arbitrary actions to after transaction close

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.