SAVE SNAPSHOT takes too much CPU
SAVE SNAPSHOT can be sped up quite a bit by avoiding costly iteration and doing I/O buffering more efficiently.
Currently it takes up to 80% CPU.
What needs to be done:
(done) - iter_inner and iter_outer removed
- kill conver_
(done) - find the optimal value for setvbuf/IO cache
- experiment with allocating index memory with MAP_SHARED flag or quickly unmmapping it after fork().
- log_io error messages are often quite obscure. At least on Linux, we can extend them
with file names (using /proc filesystem):
static const char *
nfilename(int fd)
{
#ifdef TARGET_OS_LINUX
char proc_path[32];
static __thread char filename_
sprintf(
ssize_t sz = readlink(proc_path, filename_path,
if (sz >= 0) {
return filename_path;
}
#endif /* TARGET_OS_LINUX */
return ""; /* Not implemented. */
}
- look into pipelining the write, when one thread is busy doing crc32 and formatting the data and another is writing to disk.
Each step ideally should be done separately.
Blueprint information
- Status:
- Not started
- Approver:
- None
- Priority:
- Undefined
- Drafter:
- None
- Direction:
- Needs approval
- Assignee:
- None
- Definition:
- New
- Series goal:
- None
- Implementation:
- Unknown
- Milestone target:
- None
- Started by
- Completed by