Log Archiving for XtraDB

Registered by Stewart Smith on 2011-10-04

Support log archiving.

The purpose of archiving logs is to be able to use them as a form of incremental backup.

By replaying archived logs since the last backup taken with xtrabackup, you can get an online incremental backup.

When log archiving is enabled, instead of rotating through log files, XtraDB will create a new one. There should always be one spare log file so that transactions are never stalled at log file rotation.

Vadim:
Should be it N log files YYY GB each ?
In this case we can reclaim space just be deleting old files.
Also it will be easier to ship separate files over network if needed.

Yasufumi:
"make new blank" and "switch by renaming" is the best optimized way to make archiving cost minimize.
archiving cost should be minimized. archiving process might be obstacle for progressing LSN.
Vadim, do you want to reduce the throughput or make possible stall period longer? you seem to ignore performance. Do you want just ancient method?

Blueprint information

Status:
Complete
Approver:
Alexey Kopytov
Priority:
High
Drafter:
Vadim Tkachenko
Direction:
Approved
Assignee:
Sergei Glushchenko
Definition:
Approved
Series goal:
Accepted for 5.6
Implementation:
Implemented
Milestone target:
milestone icon 5.6.11-60.3
Started by
Vadim Tkachenko on 2011-10-22
Completed by
Stewart Smith on 2013-04-03

Whiteboard

Initial implementation in
~percona-dev/percona-server/5.1.59-innodb_log_archiving
looks pretty good.

We need following improvements:

1.
set global innodb_xtra_log_archive=ON;

should be runtime variable. I want to enable / disable log archive in runtime, without server restart.

2.
innodb_xtra_log_archive=1 or ON;
both 1 and ON should be accepted.

3. Right now log name is
ib_log_archive_0000003A5A8C8600
we need decimal form instead of hex, because XtraBackup works with decimal LSN.
That is file name should be like:

ib_log_archive_000262325167341

4. I would like to have corresponding binary log position for log archive.
This will allow to have point in time recovery with binary logs.

That is when we create file
ib_log_archive_000262325167341

we also write information into 'archive_log.info', like:
262325167341:binlog.00001,23457
that is binary log position for LSN.

5. Archive log maintenance.
We need to support following option:
expire_archive_logs_sec=N

That is archive logs older than N sec should be removed automatically.

We need to support following command:

PURGE ARCHIVE LOGS { TO 'filename' | BEFORE `datetime`}
for example:
PURGE ARCHIVE LOGS TO 'ib_log_archive_000262325167341';
PURGE ARCHIVE LOGS BEFORE '2008-04-02 22:46:26';

6. Directory to keep archive logs
we need option
innodb_xtra_log_archive_dir=
to specify directory where archive logs should be stored

===========================================================================

The implementation has been rolled back to original InnoDB log archiving.
Following parameters has been introduced:
    - innodb_log_archive = ON/OFF (can be set in runtime and/or at startup)
Turns ON/OFF log archiving.

    - innodb_log_arch_dir = DIR (startup only)
Set log archiving directory (default ./ - current datadir)

    - innodb_log_arch_expire_sec = N
Number of seconds since last modification after which archived log should be deleted.

Archive names look like ib_log_archive_000262325167341 where numeric part is LSN of archive start.

Command

PURGE ARCHIVED LOGS TO 'filename';
will delete all archived logs up to 'filename' (and 'filename' too). Archive which is currently in progress will not be deleted.

PURGE ARCHIVED LOGS BEFORE datetime
will delete all archived logs modified before date time. Archive which is currently in progress will not be deleted.

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.