Priority refill for the buffer pool free list

Registered by Laurynas Biveinis on 2013-09-23

Under highly-concurrent I/O-bound workloads the following situation happens:
1) Buffer pool free lists are used faster than they are refilled by the LRU cleaner thread.
2) Buffer pool free lists become empty and more and more query and utility (i.e. purge) threads start looping in buf_LRU_get_free_block(), checking whether a free list has became non-empty, sleeping, performing single-page LRU flushes.
3) The number of free list mutex waiters increases, we have seen 200-300 query threads waiting for a free list mutex on 512-thread RW Sysbench runs. The buf_LRU_get_free_block() sleep is not effective at reducing this contention.
4) When the page cleaner thread (or a single page LRU flush by a query thread) finally produces a free page, it is starved from putting it on the free list as it must acquire the free list mutex too. However, being one thread in hundreds, the chances of a prompt acquisition are low.

The priority mutex framework is unable to fully deal with this situation: even though the cleaner thread would be high priority, so are the purge threads, which also consume free pages. Likewise the query threads are low priority, even though they would produce free pages while performing single-page flushes.

Thus:
- Adjust the free list producer, buf_LRU_block_free_non_file_page(), to always acquire the mutex with high priority.
- Adjust the free list consumer, buf_LRU_get_free_only(), to always acquire the mutex with low priority.

Ideally the setup would be three-tier, i.e. producers > utility consumers (purge) > query consumers, however such implementation would be complicated and should be done only when clearly shown that it's needed.

Blueprint information

Status:
Complete
Approver:
Alexey Kopytov
Priority:
High
Drafter:
Laurynas Biveinis
Direction:
Approved
Assignee:
Laurynas Biveinis
Definition:
Approved
Series goal:
Accepted for 5.6
Implementation:
Implemented
Milestone target:
milestone icon 5.6.13-61.0
Started by
Laurynas Biveinis on 2013-09-23
Completed by
Laurynas Biveinis on 2013-09-27

Whiteboard

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.