ThreadPool: low priority queue throttling in 5.5
One case that can limit thread pool performance and even lead to
deadlocks under high concurrency is a situation when thread groups are
oversubscribed due to active threads reaching the oversubscribe limit,
but all/most worker threads are actually waiting on locks currently held
by a transaction from another connection that is not currently in the
thread pool.
What happens in this case is that those threads in the pool that have
marked themselves inactive via thd_wait_begin() are not accounted to the
oversubscribe limit. As a result, the number of threads (both active and
waiting) in the pool grows until it hits threadpool_
connection executing the transaction which is holding the lock has
managed to enter the thread pool by then, we get a large (depending on
the threadpool_
and thus, suboptimal performance as a result. Otherwise, we get a
deadlock as no more threads can be created to process those
transaction(s) and release the lock(s).
The goal of this blueprint is to prevent such situations by throttling
the low priority queue when the total number of worker threads (both
active and waiting ones) reaches the oversubscribe limit. That is, if
there are too many worker threads, do not start new transactions and
create new threads until queued events from the already started transactions
are processed.
This rather simple change has shown a great impact on performance in our
benchmarks.
Blueprint information
- Status:
- Complete
- Approver:
- Alexey Kopytov
- Priority:
- High
- Drafter:
- Alexey Kopytov
- Direction:
- Approved
- Assignee:
- Alexey Kopytov
- Definition:
- Approved
- Series goal:
- Accepted for 5.5
- Implementation:
- Implemented
- Milestone target:
- 5.5.35-33.0
- Started by
- Alexey Kopytov
- Completed by
- Laurynas Biveinis