Server may crash due to a glibc bug in handling short-lived detached threads

Bug #1621012 reported by Laurynas Biveinis
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL Server
Unknown
Unknown
Percona Server moved to https://jira.percona.com/projects/PS
Status tracked in 5.7
5.5
Invalid
Undecided
Laurynas Biveinis
5.6
Fix Released
High
Laurynas Biveinis
5.7
Fix Released
High
Laurynas Biveinis

Bug Description

Copy of http://bugs.mysql.com/bug.php?id=82886:

[7 Sep 10:44] Laurynas Biveinis
Description:
glibc has a bug where a short-lived detached thread, that manages to complete before the caller thread finishes executing pthread_create, may crash server. In MySQL, at least the InnoDB full text parallel merge threads might be short-lived enough to hit this.

The glibc bug is https://sourceware.org/bugzilla/show_bug.cgi?id=20116 (https://sourceware.org/bugzilla/show_bug.cgi?id=19951 could be related too)

How to repeat:
Make several copies of innodb.innodb-alter testcase (move out innodb-alter-kill etc out of the way for faster repro, or just write a proper --do-test regexp). Then

$ ./mtr --debug-server --parallel=8 --do-test=innodb-alter --repeat=9000
(...)
Program terminated with signal SIGSEGV, Segmentation fault.
#0 __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:62
62 ../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x7fd4802f6700 (LWP 2706))]
#0 __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:62
#1 0x00000000008f2f47 in my_write_core (sig=sig@entry=11) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/mysys/stacktrace.c:422
#2 0x000000000066ca0c in handle_fatal_signal (sig=11) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/signal_handler.cc:236
#3 <signal handler called>
#4 __pthread_create_2_1 (newthread=newthread@entry=0x7fd4802f2268, attr=attr@entry=0x7fd4802f2270, start_routine=start_routine@entry=0x9e1350 <fts_parallel_merge(void*)>, arg=arg@entry=0x7fd4580dbe38) at pthread_create.c:713
#5 0x00000000009bb4b6 in os_thread_create_func (func=func@entry=0x9e1350 <fts_parallel_merge(void*)>, arg=arg@entry=0x7fd4580dbe38, thread_id=thread_id@entry=0x7fd4802f22e0) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/storage/innobase/os/os0thread.cc:193
#6 0x00000000009dfb04 in row_fts_start_parallel_merge (merge_info=<optimized out>) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/storage/innobase/row/row0ftsort.cc:984
#7 0x00000000009f7806 in row_merge_build_indexes (trx=0x7fd458089638, old_table=0x7fd458072798, new_table=0x7fd458072798, online=false, indexes=0x7fd4580d77a0, key_numbers=0x7fd4580d77a8, n_indexes=1, table=0x7fd4580b7700, add_cols=0x0, col_map=0x0, add_autoinc=18446744073709551615, sequence=...) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/storage/innobase/row/row0merge.cc:3742
#8 0x000000000096b673 in ha_innobase::inplace_alter_table (this=0x7fd458042680, altered_table=0x7fd4580b7700, ha_alter_info=0x7fd4802f2630) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/storage/innobase/handler/handler0alter.cc:3967
#9 0x0000000000747cad in handler::ha_inplace_alter_table (ha_alter_info=0x7fd4802f2630, altered_table=0x7fd4580b7700, this=<optimized out>) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/handler.h:2974
#10 mysql_inplace_alter_table (target_mdl_request=0x7fd4802f2710, alter_ctx=0x7fd4802f2e20, inplace_supported=HA_ALTER_INPLACE_SHARED_LOCK_AFTER_PREPARE, ha_alter_info=0x7fd4802f2630, altered_table=0x7fd4580b7700, table=0x7fd4580c62a0, table_list=0x7fd458031ab0, thd=0x1931e90) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/sql_table.cc:6850
#11 mysql_alter_table (thd=thd@entry=0x1931e90, new_db=<optimized out>, new_name=<optimized out>, create_info=create_info@entry=0x7fd4802f3fa0, table_list=table_list@entry=0x7fd458031ab0, alter_info=alter_info@entry=0x7fd4802f3ee0, order_num=0, order=0x0, ignore=false) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/sql_table.cc:8904
#12 0x000000000082954e in Sql_cmd_alter_table::execute (this=<optimized out>, thd=0x1931e90) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/sql_alter.cc:317
#13 0x00000000006f1cce in mysql_execute_command (thd=thd@entry=0x1931e90) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/sql_parse.cc:5498
#14 0x00000000006f5da8 in mysql_parse (thd=thd@entry=0x1931e90, rawbuf=<optimized out>, length=<optimized out>, parser_state=parser_state@entry=0x7fd4802f5610) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/sql_parse.cc:6972
#15 0x00000000006f70b1 in dispatch_command (command=COM_QUERY, thd=0x1931e90, packet=0x1a0daa1 "ALTER TABLE t1n ADD FULLTEXT INDEX(ct)", packet_length=<optimized out>) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/sql_parse.cc:1441
#16 0x00000000006f8fb9 in do_command (thd=<optimized out>) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/sql_parse.cc:1053
#17 0x00000000006c1722 in do_handle_one_connection (thd_arg=thd_arg@entry=0x1931e90) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/sql_connect.cc:1541
#18 0x00000000006c17c0 in handle_one_connection (arg=arg@entry=0x1931e90) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/sql/sql_connect.cc:1444
#19 0x0000000000b43856 in pfs_spawn_thread (arg=0x1984ed0) at /mnt/workspace/percona-server-5.6-trunk/BUILD_TYPE/release/Host/ubuntu-xenial-64bit/storage/perfschema/pfs.cc:1860
#20 0x00007fd4877256fa in start_thread (arg=0x7fd4802f6700) at pthread_create.c:333
#21 0x00007fd486bbab5d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:109

The crash happens in __pthread_create_2_1, after the thread has been launched, in dereferencing struct pthread *pd pointer. Strace will show its containing memory area to be munmap'ped by the spawned thread which has quit already.

Suggested fix:
Implement a workaround to pthread_join selected (only extremely short lived ones) server threads instead of pthread_detach'ing them.

With a prototype patch to do so I'm unable to reproduce the crash anymore.

tags: added: ci innodb upstream
Revision history for this message
Laurynas Biveinis (laurynas-biveinis) wrote :
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-1006

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.