Initial port to 5.7.9

Registered by Laurynas Biveinis

Merge Percona Server 5.6 to MySQL 5.7.9, producing Percona Server 5.7.9

This implements blueprint,
with user-visible changes as described at

Non-exhaustive list of non-trivial code changes in the port, core server:
- Percona Server version now consists of only single number.
- Annotate core server, MEMORY engine, and XtraDB heap memory
  allocations for Performance Schema
  (, Bump
  performance_schema_max_memory_classes so that extra PS memory
  instrumentations fit into default sizing. Bump
  performance_schema_max_rwlock_classes so that extra PS rwlock
  instrumentations still leave room for at least one of the bundled
  plugins, when loaded dynamically.
- Snapshot cloning, kill idle transaction features, thread pool, and
  to use Global_THD_manager
- Even though bug 57430 is fixed in upstream 5.7, keep our patch for
  the same, as it, combined with other TokuDB patches to prefer
  covering index accesses, still results in improved query plans.
- Removed debug-only Q_QUERY_EXEC_TIME binlog event status flag.
- Update thread pool for the connection handling refactoring of 5.7
- Added support for global backup and binlog locks for
- Per-statement variable feature parser part rewritten for the
  bottom-up parser (,,
- Utility user feature updated for auth code refactoring
- Query optimizer patches updated for,, and changes.
- Slow query log and binlog file management features have been updated
  for binlog/slow log split in the code
- Query cache patches now have search keys decoupled from its memory
  layout, QueryStripComments_Backup has been converted to LEX_CSTRING
  and made local to, and
  Query_cache::send_result_to_client no longer patches up THD fields
  in place for the query cache key computation.
- Updated code for diagnostics API changes because of
- Updated code for THD field locking changes because of
- Updated code for LOCK_thread_count split
- Updated code for MY_ATOMIC_MODE_RWLOCKS removal
- Updated code and testcase for removed INSERT DELAYED
- Cleaned up incomplete revert of our fix to bug 915814 / 64624.

- Buffer pool mutex split updated for online buffer pool resizing by
  protecting buf_pool_t::withdraw_list with free_list_mutex
- InnoDB redo log archiving feature removed due to W#8845 changing the
  log format (including clobbering the "log archived up to this LSN"
  header fields) and general lack of users of the feature.
- XtraDB fake changes prefetching feature removed.
- LRU manager thread not ported forward due to upstream multi threaded
  flusher requiring a different design for independent LRU flusher
- innodb_empty_free_list default has been changed to LEGACY as BACKOFF
  (the 5.6 default) may currently cause deadlocks due to lack of
  LRU flusher which would be separate from flush list flusher.
- Priority mutex framework not ported forward due to it needing
  updating for the upstream policy mutex
  (, and our plans to
  achieve its scalability gain differently.
- Priority rwlock framework not ported forward due to it needing
  updating for the upstream SX-lock
  (, and due to its need
  alleviated by converting the index latch to this SX-lock
- Snapshot cloning updated for 5.7 transaction management changes:
  remove the assumption on RO to RW transaction upgrade that a
  transaction always sees its own changes, which is a reasonable
  assumption in general, but is not the case for snapshot
  cloning. Also, preassign transaction ids for RO transaction views
  being cloned to use in the case of later RO to RW upgrade. Do not
  port forward the transaction descriptor
  feature. (, read_view_print
  function has been resurrected as ReadView::print(). Expand
  clone_consistent_snapshot with tests making sense in this context:
  repeated cloning of the same transaction, and cascaded cloning with
  different RO/RW transaction combinations.
- Update changed page tracking to handle new MTR types
  (,, and for different
  format/checksum algorithm
- Updated XtraDB corrupt table action feature to handle InnoDB native
  partitioning (where ha_innobase::m_share can be NULL more often than
- Make XtraDB track redo log synchronously in case of redo log
- XtraDB diagnostics were converted from fprintf to
- Updated code for innodb_use_sys_malloc removal
  with three columns PHYSICAL_PAGE_SIZE, LOGICAL_PAGE_SIZE, and
  been removed.

- Audit_log, query response time, and scalability_metrics plugins
  updated for 5.7 audit plugin API changes.
- Audit_log was converted to use MySQL plugin services for heap memory
- Plugin MTR tests were moved to plugin/tests/mtr directory from the
  main suite, effectively creating one testsuite per plugin.
- Removed HandlerSocket plugin.

- Testsuite updates to account for: MBR temp table binlogging
  differences caused by our fix for bug 1313901 / 72475; for memory to
  disk table conversion happennning at a different point due to
  combination of dynamic VARCHAR patch for MEMORY engine and bug
  (,; InnoDB made the default
  MTR engine (
- Updated partitioned InnoDB table testcases that were testing the
  partition handler to use MyISAM instead in order to avoid InnoDB
  partitioning; remove --source include/ from
  non-InnoDB-dependent testcases to implement WL#6731; converted
  testsuite to use include files for server restart/shutdown/startup
  more; added various new testcases.
- Since 5.7 removed MDL_map_partition::m_mutex, the scenario of bug
  1405076 disappears, and its testcase is removed, as debug sync
  points there do not exist anymore nor it's possible to re-introduce
- percona_expand_fast_index_creation modified to decrease not increase
  the VARCHAR size as the latter operation became online in 5.7.1.
- Removed tests that do not make sense after upstream remove
  --skip-innodb option, and some supporting code
- Removed MTR Subunit patch. Jenkins reporting will be handled externally
  to the source tree.

- Mechanical code changes: removal of HAVE_ATOMIC_BUILTINS_64,
  DYNAMIC_ARRAY in C++ code, ib_logf, HAVE_purify, UNIV_INTERN,
  replacement with ULLONGMAX, ibool/FALSE/TRUE with bool/false/true,
  string constification by using LEX_CSTRING in place of LEX_STRING

This also takes upstream fixes for the following bugs fixed in upstream
- / (MySQL shouldn't "guess" at the
variable names);
- / (handle_fatal_signal (sig=11) in
handler::ha_index_or_rnd_end - sql/handler.h:2038);
- / (ALTER TABLE crashes on corrupted
InnoDB table);
- / (mysql_upgrade binlogs queries
that fail on master);
- / (--ssl option should enforce SSL);
- / (Item_row::illegal_method_call |
void Item_row::illegal_method_call(const char*): Assertion `0' failed.).

At the same time fix
- (the
deadlock counter in PS5.6.11 is not needed) by dropping the
innodb_deadlocks status variable;
- / (Deadlocks missed by
INFORMATION_SCHEMA.INNODB_METRICS lock_deadlocks counter) by trivial
metric collection site adjustment.

Revert our features and bugfixes implemented in upstream 5.7:
- --innodb-buffer-pool-populate server option and --numa-interleave option due upstream --innodb-numa-interleave;
- super_read_only (;
- innodb_log_checksum_algorithm (75595);
- bug 1334330 / 70860;
- bug 1157037 / 68713;
- bug 1351148 / 73979;
- bug 1169494 / 68970;
- bug 1132350 / 68476;
- innodb_sched_priority_cleaner and innodb_foreground_preflush
- --secure-file-option with empty arg because of NULL arg feature;
- --innodb-log-block-size because of --innodb-log-write-ahead-size;
- all the XtraDB status counters that have enabled-by-default
  INNODB_METRICS counterparts;
- bug 1268729 / 71374;
- mysql --syslog option (;
- bug 1042517 / 66560;
- bug 1132351 / 68477;
- bug 1068210 / 67504;
- bug 606811 / 54814;
- bug 1255551 / 70854;
- slow_query_log_timestamp_precision;
- slow_query_log_timestamp_always;
- bug 1197524 / 41975;
- mysqlbinlog --rewrite-db option;
- fusionIO atomic writes;
- bug 1163439 / 68845;
- bug 1279671 / 71708;
- multiple GET_LOCK per connection feature
- bug 1238039 / 71988;
- bug 1115048 / 60682;
- bug 1331586 / 73066;
- bug 1220544 / 70228;
- max_statement_time because of max_execution_time
- Online GTID migration patch (gtid_deployment_step) because of
  dynamic gtid_mode variable
- mysqldump --ignore-create-error because of --ignore-error;
- innodb_adaptive_hash_index_partitions (62018) because of
- mutex names in SHOW ENGINE INNODB MUTEX;
- bug 1396330 / 74987;
- bug 1361568 / 73736;
- bug 1380010;
- bug 1083377 / 67685;
- bug 1262500 / 54430;
- majority of bug 878404 / 45702, except for the partial pread read
  handling in my_pread;
- error number padding patch for comp_error;
- mutex names in SHOW ENGINE INNODB MUTEX output
- mysqldump --ignore-create-error option.


Completed porting prerequisites:
- A complete 5.6 tree;
- file-ids resolved;
- 5.6 port review bugs fixed:
-- commented on;
- merge-regression bugs fixed (;
-- (Unused ha_innobase::is_corrupt() method);
-- (Duplicate LIST_PROCESS_HOST_LEN definition);
-- (Make innochecksum fully use InnoDB headers);
-- our mysql-test/include/ replaced with the upstream version.
- a 5.6/5.7 GCA branch (see the linked branches of this blueprint), check and revert patches to be dropped on that branch. 5.6.16 is fully merged to 5.7.4. 5.6.20 is fully merged to 5.7.5

Processed Jenkins:
- Do not need to define HAVE_purify anymore.

Processed features and bug fixes:
- show slave status nolock ( (reverted in the 5.6 branch)
- mysql --syslog patch (5.7.1) (reverted in the 5.6 branch)
- bug 1042517 fix (5.7.1) (reverted in the 5.6 branch)
- bug 1132351 fix (5.7.2) (reverted in the 5.6 branch. Oracle 5.7 fix is slightly different, but that's OK, PS 5.7 will be benchmarked and the fix might be adjusted as necessary).
- bug 1068210 fix (5.7.2) (reverted in the 5.6 while keeping the testcase).
- bug 606811 / bug 1186974 (5.7.4) (reverted in the 5.6 branch)
- bug 1255551 (5.7.4) (reverted in the 5.6 branch)
- slow_query_log_timestamp_precision (5.7.2) (reverted in the 5.6 branch)
- bug 1197524 fix (5.7.3) (reverted in the 5.6 branch, testcase disabled temporarily)
- mysqlbinlog --rewrite-db (5.7.2) (reverted in the 5.6 branch, testcase disabled temporarily).
- query timeout. (reverted in the 5.6 branch, will take the 5.7 implementation).
- trx descriptors: bug 1131189 / bug 1131187 / bug 1170103. (reverted in the 5.6 branch keeping the bug 1170103 testcase).
- fusionIO atomic writes (5.7.4 revs 7517, 7518) (reverted in the 5.6 branch)
- bug 1163439 fix. (reverted in the 5.6 branch)
- bug 1182535 fix (5.7.5). (reverted in the 5.6 branch. Will likely need 5.7-specific fixing later).
- NOT REVERTED bug 1279671 fix (5.7.5). Upstream fix incomplete, does not handle autoinc_mutex.
- Kostja's GET_LOCK patch (5.7.5) (reverted in the 5.6 branch).
- NOT REVERTED bug 1218347 fix (5.7.5). Upstream fix very conservative.
- bug 1238039 fix (5.7.5-) (reverted in the 5.6 branch)
- buffer pool mutex split. Sent to Oracle at, reverted in the 5.6 porting branch.
- innodb_log_block_checksum=crc32. Sent to Oracle at, reverted in the 5.6 porting branch.
- HandlerSocket plugin. (reverted in the 5.6 branch).
- fake changes. (reverted in the 5.6 branch).
- fix (reverted in the 5.6 branch)
- fix (reverted in the 5.6 branch)
- fix (reverted in the 5.6 branch, keeping the testcase)
- Subunit MTR patch (reverted in the 5.6 branch; 5.7 Jenkins does it differently).
- bug 1299688 fix (reverted in the 5.6 branch).
- removed HAVE_purify from CMake Valgrind setup
- Move plugin-specific testcases to plugin-specific test suites
- replace ibool with bool in InnoDB patch
- bug 1220544 fix / (confirmed fixed in 5.7.6)
- Removed secure_file_priv extension (keep the testcase, it tests more) (upstream implemented in 5.7.6)
- Removed innodb_sched_priority_cleaner (upstream implemented in 5.7.6)
- bug 1132350 / fix (confirmed fixed in 5.7.6)
- bug 1169494 / fix (confirmed fixed in 5.7.6).
- bug 1351148 / fix (confirmed fixed in 5.7.6).
- bug 1157037 / fix (confirmed fixed in 5.7.7).
- bug 1334330 / fix (confirmed fixed in 5.7.8).
- dropped our log block crc32 feature ( (upstream merged in 5.7.8)
- dropped our super_read_only feature (upstream implemented in 5.7.8)
- bug 1484599 / fix (confirmed fixed in 5.7.9)
- removed innodb_buffer_pool_populate and mysqld_safe numa_interleave options due to 5.7.9 innodb_numa_interleave option
- attempted to do -DWITH_UBSAN=ON testing
- removed --source include/ and ENGINE=InnoDB from non-InnoDB-specific testcases
- dropped various -master.opt options that match the changed defaults
- checked whether our testcases can use include/ and include/, include/add|, include/execute*, no_checkpoint_*, as needed
- made plugins use MySQL plugin services API
- replaced any server fprintf(stderr) with WL #6661 changes and ib::error() etc in InnoDB
- added PFS memory allocation annotations (my_malloc, ut_*nokey)
- utility user made compatible with offline_mode
- added testcases for 74900, 74584 / bug 1384656
- tested the compatibility of SHOW CREATE USER with utility user
- tested the compatibility of ACCOUNT LOCK/UNLOCK with utility user
- added testcases for 77580, 76430
- implemented and added testcase for combination of enforced and disabled storage engines


Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.


No subscribers.