SSL connection error after upgrade from 5.5.30-30.1 to 5.5.30-30.2

@Olaf,

Can you provide more details of this? I tried with the RPMs and
procedure detailed here -- https://bugs.launchpad.net/percona-server/+bug/1007164/comments/6 and it works fine.

a) Did you upgrade clients as well?

b) Is openssl installed on the system?

Hi,

- OpenSSL is installed, version 0.9.8o-4squeeze14 (Debian squeeze latest)
- I upgraded the client also (percona-server-client-5.5, 5.5.30-rel30.2-500.squeeze)
- The certificates work fine with percona server 5.5.30-30.1 so I didn't look into them. I did not change client- or server certificates in the test below

Client version 5.5.30-30.2 to server version 5.5.30-30.2:
> user@client:~# mysql -u aaa -pxxx -h server --ssl-ca=....pem --ssl-key=....key --ssl-cert=....pem
> ERROR 2026 (HY000): SSL connection error: error:00000001:lib(0):func(0):reason(1)

Client version 5.5.30-30.2 to server version 5.5.30-30.1 (same client, same certificates as before, both on server and on client):
> user@client:~# mysql -u aaa -pxxx -h server --ssl-ca=....pem --ssl-key=....key --ssl-cert=....pem
>
> Welcome to the MySQL monitor. Commands end with ; or \g.
> Server version: 5.5.30-30.1-log Percona Server (GPL), Release 30.1

I see that you tried MySQL/SP version 5.5.27 in your test? That version works fine with me as well. The problem occurred when upgrading from 5.5.30-30.1 to 5.5.30-30.2 and it happens on Debian, not RPM based.

No, I used 5.5.30.2, I pointed out that link just to highlight the procedure.

Regarding RPM v/s debian, I tried with RPM, but I will check with debian as well.

@Olaf,

I tested with both Debian and Ubuntu and I am not getting that
error.

Are you able to repeat with procedure mentioned here - https://bugs.launchpad.net/percona-server/+bug/1007164/comments/6

I tested with that.

Hi,

I tried the following:

I started with a new server install with same certificates as before: this resulted in no problems (just like you reported). At this point I started to think that it might involve some kind of ssl (shared-) library cache or something else of this kind. I upgraded everything to 5.5.30-30.2 and rebooted all machines (clients, servers). This didn't help.

Then I tried another cluster by upgrading the slaves from 5.5.30-30.1 to 5.5.30-30.2. Instantly, the clients (5.5.30.30-2) couldn't connect to the slaves anymore with the same SSL error as before.

I tried a third setup: an active-passive master-master setup, the active master (5.5.30-30.1) could not connect to the passive master after it was upgraded to 5.5.30-30.2. The error was

Last_IO_Errno: 2026
Last_IO_Error: error reconnecting to master 'username@hostname:3306' - retry-time: 60 retries: 86400

There is error 2026 (SSL connection error) again...

I've been looking into this a bit more. When connecting to localhost on a working and a non-working server, I see these differences (only different values are shown):

The '5.5.30-30.1' server (accepting ssl clients):
mysql> show status like 'ssl%';
+--------------------------------+---------+
| Variable_name | Value |
+--------------------------------+---------+
| Ssl_accepts | 0 |
| Ssl_ctx_verify_depth | 0 |
| Ssl_ctx_verify_mode | 0 |
| Ssl_finished_accepts | 0 |
| Ssl_session_cache_mode | Unknown |
| Ssl_session_cache_size | 0 |
+--------------------------------+---------+

The '5.5.30-30.2' server (not accepting ssl clients):
mysql> show status like 'ssl%';
+--------------------------------+----------------------+
| Variable_name | Value |
+--------------------------------+----------------------+
| Ssl_accepts | 626 |
| Ssl_ctx_verify_depth | 18446744073709551615 |
| Ssl_ctx_verify_mode | 5 |
| Ssl_finished_accepts | 3 |
| Ssl_session_cache_mode | SERVER |
| Ssl_session_cache_size | 128 |
+--------------------------------+----------------------+

Also, lsof on the 5.5.30-30.1 server shows

root@server:~# lsof | grep ssl | grep mysql
mysqld 27521 mysql mem REG 254,9 356608 41213 /usr/lib/libssl.so.0.9.8

while the 5.5.30-30.2 server does not show this line. I'm not sure if the server is supposed to have this shared library opened at all times?

@Olaf, thanks for the info. We will try to check with openssl
client as well.

If possible, can you try with certificates generated by openssl
(the procedure is in https://bugs.launchpad.net/percona-server/+bug/1007164/comments/6 ).

One more thing, other than downgrading you can also install/build yassl
library and LD_PRELOAD the library, that should work as well - because,
yassl also exports same symbols for compatibility.

For easily LD_PRELOAD-ing, use the option malloc-lib of
mysqld_safe - unlike what it name says, it is nothing malloc
specific, it just adds it to LD_PRELOAD.

The certificates were generated by OpenSSL

When debugging a bit more, it seems that the version of OpenSSL might be the cause of the problem. What exact version of openssl is the package built with?

@Olaf,

A)
Here are the versions:

 openssl version
 OpenSSL 0.9.8o 01 Jun 2010

B)

Regarding the 'show status' part , looks like it is working
correctly now, with correct status shown now.

C) Yes, lsof shows for me too.

sudo lsof | grep ssl | grep mysqld
mysqld 3043 mysql mem REG 253,0 380744 1059549 /usr/lib64/libssl.so.1.0.0

That should be normal.

This is with 5.5.30.2

Since you don't see it with 5.5.30.2, I wonder if it is having
issues loading the library.

Does ldd `which mysqld` | grep ssl show correctly?

Do you see any errors when running directly as:

sudo mysqld --user=mysql

(any linker errors).

The version is for debian6-64

Last night, this issue got more interesting. We upgraded a completely different cluster with clients to the Debian client version 5.5.30-2 and this time we did not touch the server (so the server is still running 5.5.30-1). Now the same issue occurs: client is unable to connect to the server with the same SSL error as before: ERROR 2026 (HY000): SSL connection error: error:00000001:lib(0):func(0):reason(1)

This suggests that not so much the server or the client is to blame, but some kind of difference between them. This still does not explain the situation where both are running the same version after an upgrade though.

We stumbled upon this bug as well after upgrading one of the slaves to Percona-Server-server-55-5.5.30-rel30.2.508.rhel5. It now can't connect to the master (Percona-Server-server-55-5.5.30-rel30.1.465.rhel6.x86_64) with exactly the same error:

ERROR 2026 (HY000): SSL connection error: error:00000001:lib(0):func(0):reason(1)

Using self-signed certificates generated by OpenSSL. They worked perfectly before.

Also it looks like Percona does not keep the previous versions of the packages in the yum repo, so we can't even downgrade back to 5.5.30-1.

figvam <email address hidden> writes:
> Also it looks like Percona does not keep the previous versions of the
> packages in the yum repo, so we can't even downgrade back to 5.5.30-1.

You can download the RPMs from the web site download area.

--
Stewart Smith

Stewart, thanks for pointing me to the right place. Downgrading to 5.5.30-1 helped to resolve the issue.

I have same problem

@Maxime, @figvam, @Olaf:

Is generating new SSL certificate with openssl and using it not working?

The procedure to generate may be slightly different since openSSL is more strict about specifications. (Let us know if certificates generated with https://bugs.launchpad.net/percona-server/+bug/1007164/comments/6 are not helping)..

@Raghavendra: I never used something else to generate certificates. My current certificates were generated with the openssl version from Debian squeeze.

@Raghavendra: same thing for me

Hate to post a "me too", but I had to abort a production upgrade because of this same problem recently:

 * Running RHEL5.9
 * Percona 5.5.30-30.1 on the masters
 * Had upgraded slaves to 5.5.30-30.2 succesfully (problem did not show up because the slaves do not have any SSL clients).
 * As soon as we upgraded the master(s) to 5.5.30-30.2, (SSL) replication to the slaves broke with the same error messages as described in this bug. Other SSL clients were no longer able to connect to the masters either. Rollback to 30.1 fixed the problem instantly.

Our setup is very simple, all certificates generated with the help of easy-rsa ( https://github.com/OpenVPN/easy-rsa ).

I wonder if this is related to http://bugs.mysql.com/bug.php?id=64461 maybe, that is, OpenSSL 0.9.8x vs OpenSSL 1.0.1y (and DHE-RSA-AES256-SHA vs DHE-RSA-AES256-SHA256) on build machine vs servers where 5.5.30-30.2 is used.

So, all reporters, please:

1. Specify the exact OpenSSL version installed on the servers affected
2. Send the output of:

ldd `which mysqld`

for 5.5.30-30.2

@Valerii:

Server version: 5.5.30-30.2-log Percona Server (GPL), Release 30.2

root@host:~$ openssl version
OpenSSL 0.9.8o 01 Jun 2010

root@host:~# ldd /usr/sbin/mysqld
 linux-vdso.so.1 => (0x00007fffd9dff000)
 libpthread.so.0 => /lib/libpthread.so.0 (0x00007fa915972000)
 libaio.so.1 => /lib/libaio.so.1 (0x00007fa915771000)
 libm.so.6 => /lib/libm.so.6 (0x00007fa9154ee000)
 librt.so.1 => /lib/librt.so.1 (0x00007fa9152e6000)
 libcrypt.so.1 => /lib/libcrypt.so.1 (0x00007fa9150af000)
 libdl.so.2 => /lib/libdl.so.2 (0x00007fa914eaa000)
 libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8 (0x00007fa914c53000)
 libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8 (0x00007fa9148b2000)
 libc.so.6 => /lib/libc.so.6 (0x00007fa91454f000)
 /lib64/ld-linux-x86-64.so.2 (0x00007fa915b95000)
 libz.so.1 => /usr/lib/libz.so.1 (0x00007fa914338000)

# cat /etc/redhat-release
CentOS release 5.9 (Final)

# rpm -q Percona-Server-server-55
Percona-Server-server-55-5.5.30-rel30.2.508.rhel5

# openssl version
OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008

# rpm -qi openssl | head -4
Name : openssl Relocations: (not relocatable)
Version : 0.9.8e Vendor: CentOS
Release : 26.el5_9.1 Build Date: Tue 05 Mar 2013 02:32:28 AM PST
Install Date: Thu 07 Mar 2013 01:31:44 PM PST Build Host: builder10.centos.org

# ldd /usr/sbin/mysqld
        linux-vdso.so.1 => (0x00007fff015fd000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00000034a5200000)
        libaio.so.1 => /usr/lib64/libaio.so.1 (0x0000003ea4200000)
        libm.so.6 => /lib64/libm.so.6 (0x00000034a5600000)
        librt.so.1 => /lib64/librt.so.1 (0x00000034a6600000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00000034a6a00000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00000034a4e00000)
        libssl.so.6 => /lib64/libssl.so.6 (0x0000003194000000)
        libcrypto.so.6 => /lib64/libcrypto.so.6 (0x00000034a7200000)
        libc.so.6 => /lib64/libc.so.6 (0x00000034a4a00000)
        /lib64/ld-linux-x86-64.so.2 (0x00000034a4600000)
        libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x0000003193c00000)
        libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x0000003193800000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x0000003193400000)
        libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00000034a9a00000)
        libz.so.1 => /lib64/libz.so.1 (0x00000034a5a00000)
        libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00000034a9e00000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00000034a8a00000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00000034a8200000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00000034a6200000)
        libsepol.so.1 => /lib64/libsepol.so.1 (0x00000034a5e00000)

For now I see that in all affected cases version 0.9.8X of OpenSSL is used. So, maybe my assumption about http://bugs.mysql.com/bug.php?id=64461 being related is not far from the reality.

Same problem here: ERROR 2026 (HY000): SSL connection error: error:00000001:lib(0):func(0):reason(1)
Digicert certs and worked fine with previous version.

#uname -a
Centos 6.4 2.6.32-358.6.2.el6.x86_64

# openssl version
OpenSSL 1.0.0-fips 29 Mar 2010

#yum list installed | grep openssl
openssl.x86_64 1.0.0-27.el6_4.2

I wonder if in cases affected server and client certificates have the same Common Name... See also probably somewhat related upstream http://bugs.mysql.com/bug.php?id=68788 (verified almost silently, so not enough public information to tell for sure).

Confirmed.
If the Common Name - the same

# mysql --ssl=1 --ssl-ca=/etc/mysql/ssl/ca-cert.pem --ssl-cert=/etc/mysql/ssl/client-cert.pem --ssl-key=/etc/mysql/ssl/client-key.pem -u root -p
Enter password:
ERROR 2026 (HY000): SSL connection error: error:00000001:lib(0):func(0):reason(1)

If different - everything is working fine

I have this issue, but I have a different CN for the client and the server certificates

So, we have 2 definite cases: one where the problem was the came CN for client and server certificates and other case where they were different but OpenSSL 0.9.8o was used.

Any connection with bug 1189460?

Roel,

As at least in one affected case this was el5:

CentOS release 5.9 (Final)

I doubt this can be directly related.

Dev, any connection with bug 1191582 ?

My guess is no connection with bug 1191582.

Here is my steps to reproduce the problem with MySQL SSL and Percona-Server-shared-compat.

[root@centos6 ~]# cat mysqli-ssl.php
<?php
$conn=mysqli_init();
mysqli_ssl_set($conn,
               '/etc/mysql-ssl/client-key.pem',
               '/etc/mysql-ssl/client-cert.pem',
               NULL, NULL, NULL);
if (!mysqli_real_connect($conn, '192.168.56.101', 'percona_ssl', '12345'))
{
    die();
}
$res = mysqli_query($conn, 'SHOW STATUS like "Ssl_cipher"');
print_r(mysqli_fetch_row($res));
mysqli_close($conn);
[root@centos6 ~]# php mysqli-ssl-test.php
PHP Warning: mysqli_real_connect(): (28000/1045): Access denied for user 'percona_ssl'@'10.38.11.76' (using password: YES) in /home/percona/mysqli-ssl-test.php on line 7

[root@centos6 ~]# rpm -e Percona-Server-shared-compat-5.5.31-rel30.3.520.rhel6.x86_64 --nodeps
[root@centos6 ~]# rpm -ivh compat-mysql51-5.1.54-1.el6.remi.x86_64.rpm
warning: compat-mysql51-5.1.54-1.el6.remi.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 00f97f56: NOKEY
Preparing... ########################################### [100%]
   1:compat-mysql51 ########################################### [100%]
[root@centos6 ~]# php mysqli-ssl-test.php
Array
(
    [0] => Ssl_cipher
    [1] => DHE-RSA-AES256-SHA
)
[root@centos6 ~]# openssl x509 -text -in /etc/mysql-ssl/client-cert.pem|grep CN
        Issuer: C=UA, ST=Lviv, L=Lviv, O=Percona, OU=Remote DBA, <email address hidden>
        Subject: C=UA, ST=Lviv, L=Lviv, O=Percona, OU=Remote DBA, <email address hidden>

The server and client certificates are generated with the different CNs. centos6.vm - server's cert CN, clone.vm - client's cert CN.
Something wrong with Percona-Server-shared-compat's libmysqlclient.so.16.
Hope this helps.

Looks like MariaDB 5.5.31 is also having similar issues: https://lists.launchpad.net/maria-discuss/msg00854.html

From my experiments, it's not about a difference between server and client certificates, but about a difference (or rather a lack of thereof) between CA cert and server cert, and between CA cert and client cert.
I recommend to those who experience the problem to run

openssl verify -CAfile <path to ca-cert.pem> <path to server cert> <path to client cert>

If you are getting "error 18 at 0 depth lookup:self signed certificate" for at least one cert, my further suggestions can be found at https://mariadb.atlassian.net/browse/MDEV-4709, specifically http://tinyurl.com/pwsp7sv.

I experience the problem, but no "error 18 at 0 depth" for me:

# openssl verify -CAfile <ca cert> <server cert> <client cert>
<server cert>: OK
<client cert>: OK

Set to confirmed based on reproduced input by Roman

All I can tell that as soon as I create certificates by the book:

http://dev.mysql.com/doc/refman/5.5/en/creating-ssl-certs.html

like this:

...
1005 openssl version
1006 mkdir cert
1007 cd cert
1008 openssl genrsa 2048 > ca-key.pem
1009 openssl req -new -x509 -nodes -days 3600 -key ca-key.pem -out ca-cert.pem
1010 openssl req -newkey rsa:2048 -days 3600 -nodes -keyout server-key.pem -out server-req.pem
1011 openssl rsa -in server-key.pem -out server-key.pem
1012 openssl x509 -req -in server-req.pem -days 3600 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 -out server-cert.pem
1013 openssl req -newkey rsa:2048 -days 3600 -nodes -keyout client-key.pem -out client-req.pem
1014 openssl rsa -in client-key.pem -out client-key.pem
1015 openssl x509 -req -in client-req.pem -days 3600 -CA ca-cert.pem -CAkey ca-key.pem -set_serial 01 -out client-cert.pem
1016 openssl verify -CAfile ca-cert.pem server-cert.pem client-cert.pem
...

everything works:

[openxs@chief msb_5_5_30]$ ./use -uroot test
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.5.30-rel30.2 Percona Server with XtraDB (GPL), Release rel30.2, Revision 500

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql [localhost] {root} (test) > grant all privileges on test.* to 'user'@'localhost' identified by 'user' require ssl;
Query OK, 0 rows affected (0.00 sec)

mysql [localhost] {root} (test) > exit
Bye
[openxs@chief msb_5_5_30]$ ./use -uuser -puser test
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 6
Server version: 5.5.30-rel30.2 Percona Server with XtraDB (GPL), Release rel30.2, Revision 500

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql [localhost] {user} (test) > select current_user();
+----------------+
| current_user() |
+----------------+
| user@localhost |
+----------------+
1 row in set (0.00 sec)

mysql [localhost] {user} (test) > show status like 'ssl_cipher';
+---------------+--------------------+
| Variable_name | Value |
+---------------+--------------------+
| Ssl_cipher | DHE-RSA-AES256-SHA |
+---------------+--------------------+
1 row in set (0.00 sec)

mysql [localhost] {user} (test) > status
--------------
/home/openxs/5.5.30/bin/mysql Ver 14.14 Distrib 5.5.30, for Linux (x86_64) using readline 5.1

Connection id: 6
Current database: test
Current user: user@localhost
SSL: Cipher in use is DHE-RSA-AES256-SHA
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server version: 5.5.30-rel30.2 P...

Actually, the problem is not mysql server (mysqld or mysql binaries) but with libmysqlclient.so.16.
According to my steps Percona-Server-shared-compat vs compat-mysql51, both have libmysqlclient.so.16 and it does not depend from libssl at all.

To test, simply take any tool (it can be my php script that uses mysqli, mysqli.so depends on .so.16 and libssl.so) that depends on .so.16, try with compat-mysql51's .so.16 and with Percona-Server-shared-compat's one.

Test #2:

Works:
[root@centos6 ~]# rpm -qa|grep compat-mysql51
compat-mysql51-5.1.54-1.el6.remi.x86_64
[root@centos6 ~]# rpm -ql compat-mysql51-5.1.54-1.el6.remi.x86_64|grep libmysqlclient.so.16
/usr/lib64/mysql/libmysqlclient.so.16
/usr/lib64/mysql/libmysqlclient.so.16.0.0
[root@centos6 ~]# ldd /usr/lib64/php/modules/mysqli.so|grep libmysqlclient.so.16
 libmysqlclient.so.16 => /usr/lib64/mysql/libmysqlclient.so.16 (0x00007fbba0af4000)
[root@centos6 ~]# php mysqli-ssl.php
Array
(
    [0] => Ssl_cipher
    [1] => DHE-RSA-AES256-SHA
)

Does not:
[root@centos6 ~]# rpm -e compat-mysql51 --nodeps
[root@centos6 ~]# yum -y install Percona-Server-shared-compat
[root@centos6 ~]# rpm -ql Percona-Server-shared-compat|grep libmysqlclient.so.16
/usr/lib64/libmysqlclient.so.16
/usr/lib64/libmysqlclient.so.16.0.0
[root@centos6 ~]# ldd /usr/lib64/php/modules/mysqli.so|grep libmysqlclient.so.16
 libmysqlclient.so.16 => /usr/lib64/libmysqlclient.so.16 (0x00007f3c831a6000)
[root@centos6 ~]# php mysqli-ssl.php
PHP Warning: mysqli_real_connect(): (28000/1045): Access denied for user 'percona_ssl'@'192.168.56.101' (using password: YES) in /root/mysqli-ssl.php on line 7
[root@centos6 ~]#

a)

I can confirm the what is said in #43. libmysqlclient.so.16.0.0
(and ones older) do not have any SSL symbols nor are linked
against libssl. This can cause any client linked against those
libraries to fail. Since the symbols don't exist in those at all (either unresolved or in the library itself) I presume they may be built without SSL.

This is an issue with packaging of shared-compat which has not
been updated for a while - https://bazaar.launchpad.net/~percona-core/percona-server/5.5/view/head:/build/percona-shared-compat.spec - version51 is still 5.1.66 there, the 5.1.70 version of Percona-Server-shared-51 (from which Percona-Server-shared-compat gets libmysqlclient.so.16.0.0) is linked correctly against libssl.

b)

However, that may be only part of the problem since mysql client
(which is linked with libssl) also seems to be affected - even
with different CNs

@Olaf: Can you confirm that even with different CNs (as per #30) you had issue
with mysql client (or was this with a client built on libmysqlclient?).

Also, the verification in #42 seems to be against openssl 1.0.x,
there also seem to be failures against openssl 0.9.8.x reported
above (and which is standard on CentOS 5.x).

The debian packages - libmysqlclient16 and libmysqlclient18 - are fine in this regard (atleast the ones on Ubuntu 12.04).

@Raghavendra: I can confirm that with different CN's I had the issue with the commandline MySQL client.

@Olaf,

Thanks for the confirmation. Also can you paste ldd for that mysql client for completeness.

@ Regarding #44,a I have opened lp:1201393 for that.

@Raghavendra:

$ ldd /usr/bin/mysql
 linux-vdso.so.1 => (0x00007fffbab88000)
 libpthread.so.0 => /lib/libpthread.so.0 (0x00007faf0c9ab000)
 libm.so.6 => /lib/libm.so.6 (0x00007faf0c729000)
 librt.so.1 => /lib/librt.so.1 (0x00007faf0c520000)
 libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8 (0x00007faf0c2c9000)
 libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8 (0x00007faf0bf28000)
 libdl.so.2 => /lib/libdl.so.2 (0x00007faf0bd23000)
 libncurses.so.5 => /lib/libncurses.so.5 (0x00007faf0badd000)
 libc.so.6 => /lib/libc.so.6 (0x00007faf0b77b000)
 /lib64/ld-linux-x86-64.so.2 (0x00007faf0cbcf000)
 libz.so.1 => /usr/lib/libz.so.1 (0x00007faf0b563000)

Hi,
just noticed this:

Percona-Server-5.5.30-rel30.1.tar.gz 2013-03-06 21.5 MB

Percona-Server-5.5.30-rel30.2.tar.gz 2013-05-14 21.5 MB

yaSSL Release notes, version 2.2.3 (04/23/2013)
"This release of yaSSL updates the test certificates as they were expired."

BUT

$ grep Release Percona-Server-5.5.30-rel30.1/extra/yassl/README | head -1
yaSSL Release notes, version 2.2.2 (7/5/2012)

$ grep Release Percona-Server-5.5.30-rel30.2/extra/yassl/README | head -1
yaSSL Release notes, version 2.2.2 (7/5/2012)

Back to yassl/certs:

$ openssl x509 -noout -in client-cert.pem -dates
notAfter=Mar 26 18:39:40 2013 GMT

$ openssl x509 -noout -in server-cert.pem -dates
notAfter=Mar 26 18:52:17 2013 GMT

$ openssl x509 -noout -in dsa-cert.pem -dates
notAfter=Mar 26 18:56:39 2013 GMT

So the Percona-Server-5.5.30-rel30.2 sources bundle a version of yassl with expired certificates, no idea on the effect of this anyway and if it can be related to this bug.

fwiw, I had this issue with upgrading from 5.5.29 to 5.5.33 in Percona-server. I was using certs that I found from 2010. I regenerated the certificates using

openssl req -x509 -days 365 -newkey rsa:2048 -keyout server-key-enc.pem -out server-cert.pem -passout pass:qwerty
openssl rsa -in server-key-enc.pem -out server-key.pem -passin pass:qwerty -passout pass:

These keys made it work. I chalk it up to these being made 3 years ago (still set to expire many years from now) from some old version of openssl.

@justinb: What version of openssl did you use? This, because I have no luck when I regenerate the certificates.

Hi,

Here are my 2 cents (with 5.1):

[not working server] # dpkg -l | grep mysqlclient
ii libmysqlclient12 4.0.24-10sarge2 mysql database client library
ii libmysqlclient15off 5.0.92-b23.87.lenny MySQL database client library
ii libmysqlclient16 5.1.72-rel14.10-597.squeeze Percona Server database client library

[working server] # dpkg -l | grep libmysql
ii libmysqlclient16 5.1.72-2 MySQL database client library

Followed by:
[not working server] # apt-get remove libmysqlclient12
[not working server] # apt-get remove libmysqlclient15off
[not working server] # apt-get install libmysqlclient16=5.1.72-2

Tadaaa! Now it works.
We had to issue:
[not working server] # echo libmysqlclient16 hold | dpkg --set-selections
in order not to be impacted by next updates if the thing isn't fixed.

Me again.

This time, same problem on servers updated to Wheezy:

# dpkg -l | grep libmysqlclient
ii libmysqlclient18 5.5.35-rel33.0-611.wheezy amd64 Percona Server database client library

# apt-cache search libmysqlclient
libglpk0 - linear programming kit with integer (MIP) support
libcrypt-mysql-perl - Perl module to emulate the MySQL PASSWORD() function
libmysqlclient-dev - MySQL database development files
libmysqlclient18 - Percona Server database client library

Ok. So how am I supposed to used another package where there aren't any others?

# apt-cache showpkg libmysqlclient18
Package: libmysqlclient18
Versions:
5.5.35-rel33.0-611.wheezy (/var/lib/dpkg/status)
 Description Language:
                 File: /var/lib/dpkg/status
                  MD5: bcc8440e50dc4f4c65b38a57b7ea31b5

5.5.33+dfsg-0+wheezy1 (/var/lib/apt/lists/security.debian.org_dists_wheezy_updates_main_binary-amd64_Packages)
 Description Language:
                 File: /var/lib/apt/lists/ftp.fr.debian.org_debian_dists_wheezy_main_binary-amd64_Packages
                  MD5: 49de807663510f5f2afd3e71efe199f1
 Description Language: en
                 File: /var/lib/apt/lists/ftp.fr.debian.org_debian_dists_wheezy_main_i18n_Translation-en
                  MD5: 49de807663510f5f2afd3e71efe199f1

5.5.31+dfsg-0+wheezy1 (/var/lib/apt/lists/ftp.fr.debian.org_debian_dists_wheezy_main_binary-amd64_Packages)
 Description Language:
                 File: /var/lib/apt/lists/ftp.fr.debian.org_debian_dists_wheezy_main_binary-amd64_Packages
                  MD5: 49de807663510f5f2afd3e71efe199f1
 Description Language: en
                 File: /var/lib/apt/lists/ftp.fr.debian.org_debian_dists_wheezy_main_i18n_Translation-en
                  MD5: 49de807663510f5f2afd3e71efe199f1
[snip]

The commend lists a lots of things, including all the available versions for the package, not only the most recent one.

# apt-get install libmysqlclient18=5.5.33+dfsg-0+wheezy1
=> downgrade to the non-percona 5.5.33 package
=> SSL connection working fine

Unfortunately I can't say what the problem is with the percona package, I can only observe that the SSL connection is working fine with the regular package.
Of course it's the exact same piece of code and .pem which are used in both cases.

Setting to Confirmed, since there is a way to reproduce it according to comment #37.

From 5.5.37 changelogs,
https://dev.mysql.com/doc/relnotes/mysql/5.5/en/news-5-5-37.html

""
 MySQL client programs from a Community Edition distribution could not connect using SSL to a MySQL server from an Enterprise Edition. This was due to a difference in certificate handling by yaSSL and OpenSSL (used for Community and Enterprise, respectively). OpenSSL expected a blank certificate to be sent when not all of the --ssl-ca, --ssl-cert, and --ssl-key options were specified, and yaSSL did not do so. To resolve this, yaSSL has been modified to send a blank certificate when an option is missing. (Bug #68788, Bug #16715064)
""

So, this issue needs to be reverified after 5.5.37 PS is
released. (5.6.17 if it applied there).

Further to comment #55 from Raghu and email from Roman (email thread 'Re: SSL connection error after upgrade'):

I guess the problem started from Percona-Server-shared-compat-5.5.30-rel30.2.509.rhel6.x86_64
Having this package or newer package installed from Percona providing libmysqlclient.so.16 breaks SSL for PHP, Python etc.

** Installing the following instead resolves the problem:
** MySQL-shared-compat-5.5.37-1.el6.x86_64

If this issue is seen after 5.5.37 PS, please post here.

Having received no comments on this bug since #56 + local testing has shown the issue to not be reproducible, I am now closing this bug, but we can always re-open it if it seen again after PS 5.5.37. Thank you for all who provided such detailed input.

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-2938