mysqld crash double free or corruption

Bug #705688 reported by Billy Becker
40
This bug affects 9 people
Affects Status Importance Assigned to Milestone
Percona Server moved to https://jira.percona.com/projects/PS
Fix Released
High
Unassigned
5.1
Fix Released
High
Unassigned
5.5
Fix Released
High
Unassigned

Bug Description

We're running Percona-Server-server-51-5.1.47-rel11.2.53.rhel5 on centos 5.5 x64

This system has 2 quad-core cpus ( xeon 5620 ) and 72GB ram

the issue happened a couple times this morning.

we're not sure which query it was running when the crash happened.

I'm attaching the portion of the error log with the backtrace for these crashes

It looks there was a double free in the QueryStripComments class cleanup method?

After the initial crash, there was another crash after it finished the recovery.

It restarted after that crash, and then it crashed again, again in the QueryStripComments cleanup method

the server started again after that crash, and we logged onto mysql and set query_cache_strip_comments to 0, and the system has been running fine since then.

Related branches

Revision history for this message
Billy Becker (billy-becker) wrote :
Revision history for this message
Stewart Smith (stewart) wrote :

Are you seeing this with current Percona Server releases? I've done a quick audit of the code and this does not currently seem possible. We can reopen this bug if you can reproduce with current Percona Server release.

Thanks for your bug report,
stewart

Changed in percona-server:
status: New → Incomplete
Revision history for this message
koyaSreekath (koya-sreekanth) wrote :

We also have same issue with this parameter and setting to query_cache_strip_comments to 0 mysql port looks stable now.

I am using Percona Server Version:
Server version: 5.1.58rel12.9-log Percona Server with XtraDB (GPL), Release rel12.9, Revision 271

Appreciate if we can get some quick feedback

thanks.
sreekanth.

Oleg Tsarev (tsarev)
Changed in percona-server:
assignee: nobody → Oleg Tsarev (tsarev)
Revision history for this message
Dr. Tyrell (dr.tyrell) wrote :

Same here, Percona-Server-server-51.x86_64 5.1.59-rel13.0.325.rhel5
I have turned off query_cache_strip_comments based on this thread and also 856404

Revision history for this message
Alexey Kopytov (akopytov) wrote :

I don't understand the status of this bug. Is this a duplicate of bug #856404? If so, this one should be marked a duplicate.

But I don't see any proof or explanations it's a duplicate. We don't even have a test case for it.

Revision history for this message
Oleg Tsarev (tsarev) wrote :

Alexey, this is another bug related to query cache enhance (also fixed in my MP).

Revision history for this message
Alexey Kopytov (akopytov) wrote :

I don't see this bug mentioned anywhere in revision comments in the fix for bug #856404.

Can we have a problem description for this bug then? Like what queries can lead to a crash under what circumstances? To make it searchable and to make sure we have this case covered in the bugfix and tests?

Revision history for this message
Oleg Tsarev (tsarev) wrote :
Revision history for this message
Alexey Kopytov (akopytov) wrote :

I would like to see an evidence this bug is actually fixed by https://code.launchpad.net/~tsarev/percona-server/5.5_fix_bug_856404/+merge/86003. Test cases, technical explanation of the root cause, whatever.

Revision history for this message
Oleg Tsarev (tsarev) wrote :

Alexey,

Than original fix was in commit 192.2.1:
    revno: 192.2.1
    fixes bug(s): https://launchpad.net/bugs/856404
    committer: Oleg Tsarev <email address hidden>
    branch nick: query_cache_strip_comments
    timestamp: Tue 2011-11-08 22:10:56 +0300
    message:
      This commit contain test and fix for buf #856404.
      While I am allocating buffer for striped query I forgot about \0 in the end of query.
      After that memory is corrupted.

@@ -108,7 +108,8 @@
 +}
 +void QueryStripComments::set(const char* query, uint query_length, uint additional_length)
 +{
-+ uint new_buffer_length = query_length + additional_length;
++ /* Don't forget about \0 in the end of query */
++ uint new_buffer_length = query_length + 1 + additional_length;
 + if(new_buffer_length > buffer_length)
 + {
 + cleanup();

Without this I receive error "double free of curruption".
After correct allocation error go out.

Do you need some another explanations?

Revision history for this message
Oleg Tsarev (tsarev) wrote :

Alexey,

Also this bug was reproduced by test in: https://code.launchpad.net/~tsarev/dbqp/dbqp_out_of_source

Revision history for this message
Laurynas Biveinis (laurynas-biveinis) wrote :

We haven't reproduced this bug on 5.1, thus marking as "Incomplete." If you experience server crashes with query cache comment stripping enabled on 5.1 (or 5.5), please reopen. Thanks!

Revision history for this message
Alexey Kopytov (akopytov) wrote :

@Alejandra Bedolla Diaz,

Please don't change the Status and Assignment fields in the Percona Server project. Those are used to track bug states in our project. If this affects another project and you want to change the state for that specific project, please use the "Also affects project" link and change the bug state there.

Revision history for this message
Laurynas Biveinis (laurynas-biveinis) wrote :

The crash was seen on 5.1, re-opening this bug for 5.1 to backport https://code.launchpad.net/~akopytov/percona-server/bug915814-5.5 revision 228 fix.

Revision history for this message
Vlad Lesin (vlad-lesin) wrote :

Laurynas, the fix for the issue was merged to 5.1 in revision number 443. The differences between Alexey's fix and fix in 443 revision are not so big and described and backported here lp:~vlad-lesin/percona-server/5.1-bug915814 .

The main issue of wrong buffer size calculating and allocating was solved in revision 443. The following change in "void QueryStripComments::set(const char* query, uint query_length, uint additional_length)" could help to avoid crashes in 5.1 tied with query_cache_strip_comments option:

@@ -466,7 +466,7 @@
         buffer[position++] = ' ';
       }
     }
- else
+ else if (query_position < query_length)
     {
       buffer[position++] = query[query_position++];
     }

Otherwise we faced with another bug.

Revision history for this message
Laurynas Biveinis (laurynas-biveinis) wrote :

Vlad, thanks, indeed there were only minor bits to backport, I was misled by the lack of a testcase, missing QUERY_BUFFER_ADDITIONAL_LENGTH macro, and because I didn't see 5.1 rev 443.

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-458

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.