Implement timeout for mysql-lvm's FLUSH TABLES WITH READ LOCK feature

Registered by Andrew Garner

flush tables with read lock can take a very long time. A timeout should be added to kill the query after a period of time. The easiest way would be via two connections - one which just sits and waits for a timeout before issuing a kill query and the other which actually issues the flush commands. I don't think there is a sane way to interrupt MySQLdb otherwise

Blueprint information

Status:
Started
Approver:
Andrew Garner
Priority:
Undefined
Drafter:
Andrew Garner
Direction:
Needs approval
Assignee:
Andrew Garner
Definition:
Drafting
Series goal:
None
Implementation:
Started
Milestone target:
None
Started by
Andrew Garner

Related branches

Sprints

Whiteboard

We currently use MySQLdb but any libmysqlclient based connector would have the same properties.

The basic problem (from a python perspective) is that all signals are trapped while we run a FLUSH TABLES WITH READ LOCK. So implementing a timeout is a little non-trivial.

I think the following approach will work well for now, and we can adjust this to a simpler mechanism if we ever move to another connector with nicer signal properties:

import threading

main_connection = MySQLdb.connect(...)
sentinel_connection = MySQLdb.connect(...)

def cancel_flush(connection, thread_id):
    cursor = connection.cursor()
    cursor.execute("KILL QUERY %d" % thread_id)
    LOG.info("Cancelled %d", thread_id)
    cursor.close()

timer = threading.timer(lvm_flush_timeout, cancel_flush, args=(sentinel_connection, main_connection.thread_id()))
timer.start()
try:
    main_connection.query('FLUSH TABLES WITH READ LOCK')
except MySQLError:
    # probably cancelled, but other errors might occur
    LOG.error(...)
    raise
else:
    timer.cancel()
    LOG.info("Lock acquired")

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.