Implement support for scaling-out multiple engines

Registered by Steven Hardy

We need support for running multiple engines for scale-out:

https://etherpad.openstack.org/heat-multiple-engines

Blueprint information

Status:
Complete
Approver:
Angus Salkeld
Priority:
High
Drafter:
None
Direction:
Approved
Assignee:
Jason Dunsmore
Definition:
Approved
Series goal:
Accepted for icehouse
Implementation:
Implemented
Milestone target:
milestone icon 2014.1
Started by
Jason Dunsmore
Completed by
Jason Dunsmore

Whiteboard

(jasond) Angus, are you working on "remove heat's watcher task"? I hope so, because I don't understand how it relates to multiple-engines ;) Here are the tests I've performed so far: http://dunsmor.com/heat/multi-engine.html Did I overlook any tests?

(asalkeld) awesome work Jason! If you created multiple autoscaling stacks then there would be duplicate checking of alarms. But I have been working at:
1) not creating that watcher_task unless really needed
2) changing over to Ceilometer (no watcher task needed)
If we stuck with that watcher task then there would be a need for figuring out what engine was responsible for what alarm checking.

(shardy) ref comments on https://review.openstack.org/#/c/43173/, bumping to heat-next

Gerrit topic: https://review.openstack.org/#q,topic:bug/1211276,n,z

Addressed by: https://review.openstack.org/43173
    Add "Action in progress" error for multi-engine

Gerrit topic: https://review.openstack.org/#q,topic:bp/multiple-engines,n,z

Addressed by: https://review.openstack.org/49440
    Implement stack-locking for multi-engine support

Addressed by: https://review.openstack.org/54190
    Ensure apt-get installs do not prompt user input

Addressed by: https://review.openstack.org/54191
    Pin version of pbr in Cloud Server resource

Addressed by: https://review.openstack.org/56080
    Utilize stack-lock for multi-engine support

Addressed by: https://review.openstack.org/56476
    Database changes for multi-engine support

Addressed by: https://review.openstack.org/56477
    Implement stack-locking for multi-engine support

Addressed by: https://review.openstack.org/59851
    Implement engine listener service for multi-engine support

(?)

Work Items

Work items:
add alarming to ceilometer: INPROGRESS
remove heat's watcher task: TODO
change the RPC to be round robin: DONE
add a locking mechanism so multiple stack ops don't happen at the same time: DONE

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.