Reload configuration files on SIGHUP signal

Registered by Abhishek Kekane

In production environment, administrator modifies the glance-api.conf configuration parameters like filesystem_store_datadirs when the storage is almost full to add more capacity by adding more disks, or to increase the number of workers or log configuration etc. then he/she need to restart the glance service explicitly to make these changes effective. Restarting service would break users connected to it which is not good in users point of view.

Add the ability to dynamically change configuration settings of a running glance server with no impact to service.

A running glance server consists of a parent/master process and one or more child/worker processes.

On receipt of a SIGHUP signal the master process will:

- reload the configuration
- send a SIGHUP to the original workers
- start new workers with the new configuration
- its listening socket will not be closed

On receipt of a SIGHUP signal each original worker process will:

- close the listening socket so as not to accept new requests
- complete any in-flight requests
- exit

This approach is based on nginx's behaviour and avoids some of the disadvantages of the current oslo's Launcher reload:

- Race conditions: Launcher does not shutdown eventlet cleanly, existing
  requests can fail.
- If all workers are busy there can be a lengthy delay when new requests
  are not processed.
- Long lived pre-SIGHUP idle client connections can stall request
  processing indefinitely.
- Not all parameters can be changed, eg number of workers.
- The wsgi pipeline cannot be changed, eg to enable caching.

Note:
The following eventlet change is required:
https://github.com/eventlet/eventlet/pull/138/files

Blueprint information

Status:
Complete
Approver:
Nikhil Komawar
Priority:
Medium
Drafter:
Abhishek Kekane
Direction:
Approved
Assignee:
Stuart McLaren
Definition:
Approved
Series goal:
Accepted for kilo
Implementation:
Implemented
Milestone target:
milestone icon 2015.1.0
Started by
Nikhil Komawar
Completed by
Nikhil Komawar

Related branches

Sprints

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.