redis requirepass problem

Bug #1708376 reported by jian.song
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack DBaaS (Trove)
Fix Released
Undecided
jian.song

Bug Description

When we have a normal operation of the redis, through the configuration to add a requirepass time, because you we not use password, then can not complete the ping operation [1]. At this time, the instance state will be set to shutdown. There are two solutions, 1: capture exception information, when the abnormal information is displayed because of the authority of the time, set to active. 2 think of ways to create a password with the client to perform the ping operation

[1]:2017-08-03 06:55:56.934 ERROR trove.guestagent.datastore.experimental.redis.service [^[[00;36m-] ^[[01;35mError getting Redis status.^[[00m
2017-08-03 06:55:56.934 TRACE trove.guestagent.datastore.experimental.redis.service ^[[01;35m^[[00mTraceback (most recent call last):
2017-08-03 06:55:56.934 TRACE trove.guestagent.datastore.experimental.redis.service ^[[01;35m^[[00m File "/root/trove/trove/trove/guestagent/datastore/experimental/redis/service.py", line 57, in _get_actual_db_status
2017-08-03 06:55:56.934 TRACE trove.guestagent.datastore.experimental.redis.service ^[[01;35m^[[00m if self.__client.ping():
2017-08-03 06:55:56.934 TRACE trove.guestagent.datastore.experimental.redis.service ^[[01;35m^[[00m File "/root/trove/trove/trove/guestagent/datastore/experimental/redis/service.py", line 418, in ping
2017-08-03 06:55:56.934 TRACE trove.guestagent.datastore.experimental.redis.service ^[[01;35m^[[00m return self.__client.ping()
2017-08-03 06:55:56.934 TRACE trove.guestagent.datastore.experimental.redis.service ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 682, in ping
2017-08-03 06:55:56.934 TRACE trove.guestagent.datastore.experimental.redis.service ^[[01;35m^[[00m return self.execute_command('PING')
2017-08-03 06:55:56.934 TRACE trove.guestagent.datastore.experimental.redis.service ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 573, in execute_command
2017-08-03 06:55:56.934 TRACE trove.guestagent.datastore.experimental.redis.service ^[[01;35m^[[00m return self.parse_response(connection, command_name, **options)
2017-08-03 06:55:56.934 TRACE trove.guestagent.datastore.experimental.redis.service ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 585, in parse_response
2017-08-03 06:55:56.934 TRACE trove.guestagent.datastore.experimental.redis.service ^[[01;35m^[[00m response = connection.read_response()
2017-08-03 06:55:56.934 TRACE trove.guestagent.datastore.experimental.redis.service ^[[01;35m^[[00m File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 582, in read_response
2017-08-03 06:55:56.934 TRACE trove.guestagent.datastore.experimental.redis.service ^[[01;35m^[[00m raise response
2017-08-03 06:55:56.934 TRACE trove.guestagent.datastore.experimental.redis.service ^[[01;35m^[[00mResponseError: NOAUTH Authentication required.

Amrith Kumar (amrith)
Changed in trove:
status: New → Incomplete
Revision history for this message
jian.song (jiansong) wrote :

Hi, amrith, this is a really problem, want to hear your opinion.

Revision history for this message
Yiu-Chung Lee (lee-yiu-chung) wrote :

I have identical problem. The problem here is redis configuration and redis connection is made is read BEFORE the requirepass is set.

I workarounded this bug by re-reading configuration and re-initialize redis connection after requirepass is set, but it is really crude and hackish...

Revision history for this message
jian.song (jiansong) wrote :

thx Yiu,this is a way to solve this problem,and I test in my enviorments.over time, I will see if there is no better way.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to trove (master)

Fix proposed to branch: master
Review: https://review.openstack.org/514955

Changed in trove:
assignee: nobody → jian.song (jiansong)
status: Incomplete → In Progress
Revision history for this message
jian.song (jiansong) wrote :

hello, Yiu-Chung Lee I tried a new solution. I have observed that the write requirepass before the restart. So we need to ensure that, from this instance to start to change the configuration to close the database before the use of a client. (For example, when we write this parameter for the first time, we should ensure that the client is shutting down the database when the restart, the client is without requirepass, otherwise it will be error.) But we also have to do before the restart, Let client require require so that client can ping. I put the generic restart method to get the redis guestagent to rewrite, beaten for this purpose. Please see my changes.

https://review.openstack.org/#/c/514955/

If you can, you can provide your community account, I can add you co-author-by.

Revision history for this message
Fan Zhang (fanzhang) wrote :

My solution is that rebuild admin client inside trove.guestagent.datastore.experimental.redis.service.RedisApp#apply_overrides, since using configuration group to enable redis authentication may call this method(actually, I'm not sure). Here is my solution:
```
...
REBUILT_NEEDED_CONFIGS = [u'requirepass', ]
...
    def apply_overrides(self, client, overrides):
        for prop_name, prop_args in overrides.items():
            args_string = self._join_lists(
                self._value_converter.to_strings(prop_args), ' ')
            client.config_set(prop_name, args_string)
            if prop_name in REBUILT_NEEDED_CONFIGS:
                self._rebuild_admin_client()
                client = self.admin
```

In my implementation of redis root-enable, I have to rebuild admin client to get CONFIG SET rename, also it can solve this problem I think. I am going to organize my code and then commit some patch sets for this bp[1]. Specs about this bp is here [2]. That would be great if you could help review this bp and spec.

[1]https://blueprints.launchpad.net/trove/+spec/root-enable-in-redis
[2]https://review.openstack.org/#/c/514940/

Revision history for this message
Fan Zhang (fanzhang) wrote :

And _rebuild_admin_client() goes here:
```
    def _rebuild_admin_client(self):
        self.admin = self._build_admin_client()
        self.status.set_client(self.admin)
```

Revision history for this message
Fan Zhang (fanzhang) wrote :

I think my problem here is different from yours. So, just as a conference, maybe we can work on update_overrides or somewhere rather than inside restart.

Revision history for this message
jian.song (jiansong) wrote :

Yes,I read it, requirepass does not need to be restarted. I entered a wrong idea because some location causes the mount to fail...It seems that we need to add update client in apply_override and update_override. (should be),need to continue to investigate

Revision history for this message
Fan Zhang (fanzhang) wrote :

I met this problem after I updated redis.conf without using command config set requirepass, and then restarted redis service. So I thought it would be perfect to rebuild admin client inside apply_overrides. It was valuable rebuilding
admin client to avoid heartbeat failure, which caused redis instance status setting to SHUTDOWN.

I look through the code about enabling requirepass via configuration group, it seems that update_overrides is what it uses.

Further investigation sounds great.

Revision history for this message
jian.song (jiansong) wrote :

I tried it, it seems to be first update_override, and then through the apply_override to complete the configuration of the write. apply_override where it looks like the best choice.

Well, I've seen you about the root-related spec. In fact, this is a very good proposal, but I think your spec write and I often seem not the same, not sure whether this format is complete.

In addition, I think we are now the primary task is to complete the reconstruction of this problem Adminclient. Then we can be prepared to increase the root-related functions.

My previous fix is ​​a failure, so I would like to ask your opinion, I hope to be able to come up with enough to fix the bug code. And then wait for your spec to be added after the other.

By the way,I feel that part of your code is redundant....

...
    def apply_overrides (self, client, overrides):
        for prop_name, prop_args in overrides.items ():
            args_string = self._join_lists (
                self._value_converter.to_strings (prop_args), '')
            client.config_set (prop_name, args_string)
            self.admin = self._build_admin_client ()
            self.status = RedisAppStatus (self.admin)
`` ``

It seems that this can be done, I think we can rebuild the client every time.
Of course it might take into account your changes because of your spec.

Revision history for this message
jian.song (jiansong) wrote :

seems client from here,so we can just update self.admin

   def apply_overrides(self, context, overrides):
        LOG.debug("Applying overrides.")
        self._app.apply_overrides(self._app.admin, overrides)

Revision history for this message
Fan Zhang (fanzhang) wrote :

I am not sure that it's appropriate to rebuild admin client every time after applying overrides. But it seems like
a solution to solve the problem. I think you can commit a patch for review.We'll see then.

As for the specs format, I used template.rst, and consulted some specs in ocata. But if someone could point the incomplete parts of this spec, I'd like to work on it.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to trove (master)

Reviewed: https://review.openstack.org/514955
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=0a8f5c31bb2f3786ddff2800762dbc06a68a1397
Submitter: Zuul
Branch: master

commit 0a8f5c31bb2f3786ddff2800762dbc06a68a1397
Author: jiansong <email address hidden>
Date: Wed Oct 25 20:12:31 2017 -0700

    Fix requirepass problem with redis

    When we use configuration to configure the request for the redis
    requirepass, Since we did not rebuild the adminclient with
    password, the previous client did not have permission to ping.
    Will cause the status to enter shutdown. It is now modified to
    rebuild the client as long as the configuration changes are made

    co-author-by:Fan Zhang <email address hidden>
    Change-Id: I9a17eaf2b69da0571295b38c5b2ec87dbc30e1d9
    Closes-Bug: #1708376

Changed in trove:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/trove 9.0.0.0b3

This issue was fixed in the openstack/trove 9.0.0.0b3 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.