Pluggable transports for worker-based engine

Registered by Ivan Melnikov on 2014-03-31

Instead of using kombu and only kombu for communication between engine and workers, we should use pluggable mechanism that should abstract away communication protocol.

Blueprint information

Status:
Not started
Approver:
Joshua Harlow
Priority:
Medium
Drafter:
Ivan Melnikov
Direction:
Approved
Assignee:
None
Definition:
Approved
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

Possible backends include:
- kombu (convert current code to be a backend)
- oslo.messaging (really, we should support it)
- http/rest
  - websockets? engine polling for result? workers polling for tasks?
- zookeeper
  - we love it =)
  - re-use some of jobboard code?

I (iv_m) think that these transport can operate on higher level of abstraction (e.g. know about tasks and flows and everything), so transport implemented this way might be simpler and generally better for our tasks than e.g. generic kombu virtual transport or oslo.messaging backend using same technologies.

Another option would be to switch to oslo.messaging completely and (possibly) implement some of backends as oslo.messaging backends.

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.