Too easy to backup wrong datadir with multiple instances

Bug #1343722 reported by MikeG
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona XtraBackup moved to https://jira.percona.com/projects/PXB
Fix Released
Wishlist
Alexey Kopytov
2.1
Won't Fix
Undecided
Unassigned
2.2
Fix Released
Wishlist
Alexey Kopytov

Bug Description

If you have a valid datadir that is referenced in /etc/my.cnf and another instance you specify by socket, you can silently back up the wrong directory when you forget to pass --defaults-file:

# innobackupex --socket=/s/56master/data/mysql.sock --stream=tar . > /dev/null

<SNIP>

140716 19:05:51 innobackupex: Starting to backup non-InnoDB tables and files
innobackupex: in subdirectories of '/var/lib/mysql'
innobackupex: Backing up files '/var/lib/mysql/test/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (28 files)

# mysql -S/s/56master/data/mysql.sock -e'select @@datadir'
/s/56master/data/

A straight forward fix would be to compare $orig_datadir/ to @@datadir that you SELECT over the socket:

For percona-xtrabackup-2.2.3-4982.el6.x86_64:

@@ -1899,6 +1899,13 @@
     my $buffer_pool_filename = get_option(\%config, $option_defaults_group,
                                           'innodb_buffer_pool_filename');

+ # Validate datadir for current socket before doing any heavy lifting
+ my ($datadir_var) = eval { $mysql{dbh}->selectrow_array("SELECT \@\@datadir") };
+ if ( "$orig_datadir/" ne "$datadir_var" ) {
+ print STDERR "$prefix Error: $orig_datadir incorrect for this unix socket\n";
+ exit 1;
+ }
+
     detect_mysql_capabilities_for_backup(\%mysql);

     #

For percona-xtrabackup-2.1.9-744.rhel6.x86_64 it is quite similar @@ -1916,6 +1916,22 @@

If this is not acceptable or conflicts with goals in https://blueprints.launchpad.net/percona-xtrabackup/+spec/read-server-options-with-show-variables it might be ok to create a view and then search for it:

     my $buffer_pool_filename = get_option(\%config, $option_defaults_group,
                                           'innodb_buffer_pool_filename');

+ # Validate datadir for current socket before doing any heavy lifting
+ use Digest::MD5 qw(md5_hex);
+ my $test_view = md5_hex("$now $prefix $orig_datadir"); # Avoid replication race
+ print STDERR "\n";
+ print STDERR "$now $prefix Creating a test view to validate that datadir is $orig_datadir\n";
+ mysql_query(\%mysql, "CREATE VIEW mysql.xtrabackup_$test_view AS SELECT 'xtrabackup_test_view'");
+ if (!-f "$orig_datadir/mysql/xtrabackup_$test_view.frm" ) {
+ print STDERR "$now $prefix Dropping test view\n";
+ mysql_query(\%mysql, "DROP VIEW mysql.xtrabackup_$test_view");
+ print STDERR "Detected $orig_datadir to be incorrect for this unix socket\n";
+ exit 1;
+ } else {
+ print STDERR "$now $prefix Datadir validated, dropping test view\n";
+ mysql_query(\%mysql, "DROP VIEW mysql.xtrabackup_$test_view");
+ }
+
     detect_mysql_capabilities_for_backup(\%mysql);

     #

At least one problem with the heavy handed approach is that there is a race condition if the slave runs a backup at the same second but server_id does not seem to be available via xtrabackup --print-param. Perhaps it could be extended to toggle the session variable for sql_log_bin/wsrep_on to avoid this. Another potential problem is that a "read only" tool uses DDL.

Related branches

Revision history for this message
MikeG (mikegriffin) wrote :

This might also occur without multiple instances, if you have changed datadir in /etc/my.cnf but have not yet restarted the server.

Revision history for this message
Nilnandan Joshi (nilnandan-joshi) wrote :

Easy to reproducible.

nilnandan@nilnandan-Dell-XPS:~$ mysql --socket=/tmp/mysql_sandbox5615.sock -e "select @@datadir"
+--------------------------------------------+
| @@datadir |
+--------------------------------------------+
| /home/nilnandan/sandboxes/msb_5_6_15/data/ |
+--------------------------------------------+

nilnandan@nilnandan-Dell-XPS:~$ sudo innobackupex --socket=/tmp/mysql_sandbox5615.sock /home/nilnandan/backup

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

...
xtrabackup version 2.2.3 based on MySQL server 5.6.17 Linux (x86_64) (revision id: )
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
...
innobackupex: in subdirectories of '/var/lib/mysql'
innobackupex: Backing up files '/var/lib/mysql/mysql/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (74 files)
>> log scanned up to (3171230103)
innobackupex: Backing up files '/var/lib/mysql/performance_schema/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (53 files)
innobackupex: Backing up file '/var/lib/mysql/dbtest/db.opt'
innobackupex: Backing up file '/var/lib/mysql/dbtest/sbtest.frm'
innobackupex: Backing up files '/var/lib/mysql/test/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (10 files)
140723 08:58:33 innobackupex: Finished backing up non-InnoDB tables and files

140723 08:58:33 innobackupex: Executing FLUSH ENGINE LOGS...
140723 08:58:33 innobackupex: Waiting for log copying to finish

>> log scanned up to (3171230103)
xtrabackup: The latest check point (for incremental): '3171230103'
xtrabackup: Stopping log copying thread.
.>> log scanned up to (3171230103)

xtrabackup: Creating suspend file '/home/nilnandan/backup/2014-07-23_08-57-58/xtrabackup_log_copied' with pid '3297'
xtrabackup: Transaction log of lsn (3171230103) to (3171230103) was copied.
140723 08:58:34 innobackupex: All tables unlocked

innobackupex: Backup created in directory '/home/nilnandan/backup/2014-07-23_08-57-58'
140723 08:58:34 innobackupex: Connection to database server closed
140723 08:58:34 innobackupex: completed OK!
nilnandan@nilnandan-Dell-XPS:~$

Changed in percona-xtrabackup:
status: New → Confirmed
Revision history for this message
Alexey Kopytov (akopytov) wrote :

This looks like another issue that would be automatically fixed by https://blueprints.launchpad.net/percona-xtrabackup/+spec/read-server-options-with-show-variables. I don't see any conflicts here.

Revision history for this message
MikeG (mikegriffin) wrote :

This seems to be fixed as of 2.2.6:

"""
Percona XtraBackup now reads server options from SHOW VARIABLES rather than my.cnf configuration file
"""

In testing, the backup rightfully fails with the following error:

[ERROR] ! innobackupex: Error: option 'datadir' has different values:
[ERROR] ! '/stage/mysqlstage/data' in defaults file
[ERROR] ! '/live/mysqllive/data/' in SHOW VARIABLES

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-1009

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.