Dynamically obtain addresses and statuses from workers for worker-based engine

Registered by Ivan Melnikov on 2014-03-31

Worker-based engine should be able to dynamically obtain information about workers, inluding:
- service discovery: where the workers are available and what each of them can do
- status: what is each worker doing (which tasks are running on it right now)

This should be done in a way that does not affect, or better improves scalability and resilience of WBE.

It is suggested that this functionality should be implemented in pluggable manner, allowing engine to employ one of provided backends through common interface.

Blueprint information

Status:
Started
Approver:
Joshua Harlow
Priority:
High
Drafter:
Ivan Melnikov
Direction:
Approved
Assignee:
Joshua Harlow
Definition:
Approved
Series goal:
None
Implementation:
Slow progress
Milestone target:
None
Started by
Joshua Harlow on 2014-04-10

Related branches

Sprints

Whiteboard

The possible backends for this may use:
- same transport that is used for communication for transports that support broadcasts/fanouts;
- fixed configuration (e.g. have one backend that works the way engine works now: gets list of topics from config instead of querying it somewhere);
- zookeeper;
- filesystem
- ...

Gerrit topic: https://review.openstack.org/#q,topic:bp/wbe-worker-info,n,z

Addressed by: https://review.openstack.org/91509
    Working on wbe-worker-info (WIP)

Addressed by: https://review.openstack.org/150253
    Begin abstracting out the topic requesting from WBE engine

Addressed by: https://review.openstack.org/144212
    Use monotonic time when/if available

Addressed by: https://review.openstack.org/151495
    WIP for having the ability to have tooz find workers+topics

Addressed by: https://review.openstack.org/151911
    Add a WBE worker finder base class and use it

Addressed by: https://review.openstack.org/153111
    Use a notifier instead of a direct property assignment

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.