Graceful service shutdown

Registered by Brian Elliott on 2013-03-01

Enable Nova services to be terminated gracefully. Disable processing of new requests, but allow new requests to complete before terminating the process.

During a software upgrade, this would allow service instances to be swapped out, while still completing existing requests.

Steps:
1) Disable message "listening".
2) Disable periodic task timer.
3) Wait for existing requests and periodic tasks to complete.
4) Kill process.

Blueprint information

Status:
Complete
Approver:
Russell Bryant
Priority:
Low
Drafter:
Brian Elliott
Direction:
Approved
Assignee:
Christopher Lefelhocz
Definition:
Approved
Series goal:
Accepted for icehouse
Implementation:
Implemented
Milestone target:
milestone icon 2014.1
Started by
Thierry Carrez on 2013-05-14
Completed by
Russell Bryant on 2013-11-15

Related branches

Sprints

Whiteboard

Havana summit etherpad: https://etherpad.openstack.org/ZeroDowntimeUpgrades

Gerrit topic: https://review.openstack.org/#q,topic:bp/graceful-shutdown,n,z

Addressed by: https://review.openstack.org/28990
    Make launchers trigger service cleanup on shutdown

related review for oslo: https://review.openstack.org/#/c/30793/

bug related to that oslo change: https://bugs.launchpad.net/oslo/+bug/1199315

Addressed by: https://review.openstack.org/47174
    graceful-shutdown: add graceful shutdown into compute

(?)

Work Items

Work items:
Modify launchers to initiate a graceful shutdown when SIGTERM/SIGINT is raised. (Right now the service greenthread is simply killed.): DONE
Modify "Consumers" stop receiving RPC messages: DONE
Terminate RPC threads: DONE
Disable periodic task looping: DONE
Disable service group thread: DONE
Make shutdown wait for in-progress work to finish: DONE

Work items for icehouse-1:
Port Oslo code to nova: INPROGRESS

Dependency tree

* Blueprints in grey have been implemented.