dialog.so cannot be loaded with perl and python clients when plugin-dir is set on [client] section of my.cnf

Assuming this is set on my.cnf:

[client]
plugin-dir=/usr/lib64/mysql/plugin

On Perl Client, with source code:
use DBI;
 $dbh = DBI->connect('dbi:mysql:mysql;mysql_read_default_file=/etc/my.cnf','ap','123')
 or die "Connection Error: $DBI::errstr\n";
 $sql = "select * from mysql.user";
 $sth = $dbh->prepare($sql);
 $sth->execute
 or die "SQL Error: $DBI::errstr\n";
 while (@row = $sth->fetchrow_array) {
 print "@row\n";
 }

This will error out as:
DBI connect('mysql;mysql_read_default_file=/etc/my.cnf','ap',...) failed: Authentication plugin 'dialog' cannot be loaded: /dialog.so: cannot open shared object file: No such file or directory at mysqltest.pl line 2
Connection Error: Authentication plugin 'dialog' cannot be loaded: /dialog.so: cannot open shared object file: No such file or directory

However, if you copy dialog.so to parent directory(/), it will work.

On Python client, with source code:
import MySQLdb
db = MySQLdb.connect(host="localhost", user="ap", passwd="123", db="mysql",read_default_file="/etc/my.cnf")

This will error out as:
  File "test.py", line 3, in <module>
    db = MySQLdb.connect(host="localhost", user="ap", passwd="123", db="mysql",read_default_file="/etc/my.cnf")
  File "/usr/lib64/python2.6/site-packages/MySQL_python-1.2.4b4-py2.6-linux-x86_64.egg/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/MySQL_python-1.2.4b4-py2.6-linux-x86_64.egg/MySQLdb/connections.py", line 187, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (2059, "Authentication plugin 'dialog' cannot be loaded: \xf5\xae\xc43\xef\x7f/dialog.so: cannot open shared object file: No such file or directory")

If installation is based on RPM, say for CentOS 6, you can comment out plugin-dir in my.cnf file and authentication will work.

However, the issue is dialog.so won't be loaded if Percona Server is installed using the binary version and placing it on a custom directory.