MyRocks and TokuDB common macros and build with FORCE option

Registered by George Ormond Lorch III on 2017-04-06

- Implement common CMake macros in server tree for determining cxx11 and other common environmental support.

-Alter -DWITH_(TOKUDB | ROCKSDB) to be tristate, 0|OFF|off to skip engine, 1|ON|on to try to build but fail gracefully if unsupported environment, or FORCE to try to build but fail with error if unsupported environment.

see https://jira.percona.com/browse/MYR-38

Blueprint information

Status:
Complete
Approver:
George Ormond Lorch III
Priority:
High
Drafter:
George Ormond Lorch III
Direction:
Approved
Assignee:
George Ormond Lorch III
Definition:
Approved
Series goal:
Accepted for 5.7
Implementation:
Implemented
Milestone target:
milestone icon 5.7.18-14
Started by
George Ormond Lorch III on 2017-04-06
Completed by
George Ormond Lorch III on 2017-04-07

Related branches

Sprints

Whiteboard

Jenkins jobs, all just run some common named test from tokudb and rocksdb suites as we don't need total mtr here:
--suite=tokudb,rocksdb,innodb --do-test=gap_lock_error

runs with all default cmake options
-----------------------------------
http://jenkins.percona.com/view/5.7/job/mysql-5.7-param/796/

We can see that all platforms pass:
 - None build MyRocks by default
 - centos 6 32 skips TokuDB as an unsupported platform
   - 19:55:50 -- Not building with MyRocks storage engine, you must specify -DWITH_ROCKSDB=1|ON|FORCE to cmake in order to build with MyRocks.
   - 19:55:50 -- Building with TokuDB storage engine = 1
   - 19:55:50 CMake Warning at storage/tokudb/CMakeLists.txt:43 (MESSAGE):
   - 19:55:50 x86_64 is only platform supported. i686 found. Not building TokuDB
 - remaining platforms build TokuDB and pass tests

-DWITH_TOKUDB=0 -DWITH_ROCKSDB=0
--------------------------------
http://jenkins.percona.com/view/5.7/job/mysql-5.7-param/797/
http://jenkins.percona.com/view/5.7/job/mysql-5.7-param/800/

There was a jenkins/github issues, so needed a matrix reload.
We can see again all platforms pass:
 - None build MyRocks or TokuDB at all and all have the following warnings from cmake
   - 20:01:48 -- Not building with TokuDB storage engine, you must specify -DWITH_TOKUDB=1|ON|FORCE to cmake in order to build with TokuDB.
   - 20:01:48 -- Not building with MyRocks storage engine, you must specify -DWITH_ROCKSDB=1|ON|FORCE to cmake in order to build with MyRocks.

-DWITH_TOKUDB=1 -DWITH_ROCKSDB=1
--------------------------------
http://jenkins.percona.com/view/5.7/job/mysql-5.7-param/798/
http://jenkins.percona.com/view/5.7/job/mysql-5.7-param/801/

There was a jenkins/github issues, so needed a matrix reload.

We can see again all platforms pass except for yakkety debug which is due to a compilation issue in MyRocks, logged ans a pretty easy fix:
 - 21:12:43 /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-yakkety-64bit/storage/rocksdb/ha_rocksdb.cc: In member function ‘void myrocks::ha_rocksdb::convert_record_to_storage_format(const rocksdb::Slice&, myrocks::Rdb_string_writer*, rocksdb::Slice*)’:
  - 21:12:43 /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-yakkety-64bit/storage/rocksdb/./././rdb_utils.h:97:48: error: nonnull argument ‘pk_unpack_info’ compared to NULL [-Werror=nonnull-compare]

This is where the changes get more interesting as MyRocks is now compiling and testing where is should, and not where it shouldn't:
 - centos 6 32:
   - 20:03:55 -- Building with MyRocks storage engine = 1
   - 20:03:55 CMake Warning at storage/rocksdb/CMakeLists.txt:22 (MESSAGE):
   - 20:03:55 x86_64 is only platform supported. i686 found. Not building MyRocks
 - debian wheezy:
   - 20:04:26 CMake Warning at storage/rocksdb/CMakeLists.txt:29 (MESSAGE):
   - 20:04:26 GCC >= 4.8.0 required. 4.7.2 found. Not building MyRocks
 - all others build and test with noted exception of yakkety debug
   - 20:01:12 -- Building with MyRocks storage engine = 1
   ...
   - 20:37:36 ==============================================================================
   - 20:37:36
   - 20:37:36 TEST WORKER RESULT TIME (ms) or COMMENT
   - 20:37:36 --------------------------------------------------------------------------
   - 20:37:36
   - 20:37:36 worker[3] Using MTR_BUILD_THREAD 301, with reserved ports 13010..13019
   - 20:37:36 worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 13000..13009
   - 20:37:36 worker[2] Using MTR_BUILD_THREAD 302, with reserved ports 13020..13029
   - 20:37:37 rocksdb.gap_lock_error w3 [ pass ] 502
   - 20:37:37 innodb.gap_lock_error w1 [ pass ] 488
   - 20:37:37 tokudb.gap_lock_error w2 [ pass ] 529
   - 20:37:41 --------------------------------------------------------------------------
   - 20:37:41 The servers were restarted 0 times
   - 20:37:41 Spent 1.519 of 8 seconds executing testcases
   - 20:37:41
   - 20:37:41 Completed: All 3 tests were successful.

-DWITH_TOKUDB=FORCE -DWITH_ROCKSDB=FORCE
----------------------------------------
http://jenkins.percona.com/view/5.7/job/mysql-5.7-param/799/
http://jenkins.percona.com/view/5.7/job/mysql-5.7-param/802/

There was a jenkins/github issues, so needed a matrix reload.

In this test, we are forcefully building TokuDB and MyRocks on all platforms, those where they are no supported show as complete cmake/build failures:
 - centos 6 32:
   - 20:07:02 CMake Error at storage/rocksdb/CMakeLists.txt:22 (MESSAGE):
   - 20:07:02 x86_64 is only platform supported. i686 found. Not building MyRocks
 - debian wheezy:
   - 20:18:54 CMake Error at storage/rocksdb/CMakeLists.txt:29 (MESSAGE):
   - 20:18:54 GCC >= 4.8.0 required. 4.7.2 found. Not building MyRocks
 - all others build and test as above still with exception of yakkety debug

Summary:
--------
These changes 'normalize' the cmake checks between MyRocks and TokuDB and with the exception of the yakkety compilation failre, will allow TokuDB and MyRocks to build cleanly on all supported platforms and skip those where they are not supported without needing any external scripting to exclude the platforms. The addition of the FORCE allows release builder scripts to ensure they are built fully on desired platforms or error out at the cmake stage.

Tracking https://jira.percona.com/browse/BLD-691

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.