Pluggable transports for worker-based engine
Instead of using kombu and only kombu for communication between engine and workers, we should use pluggable mechanism that should abstract away communication protocol.
Blueprint information
- Status:
- Not started
- Approver:
- Joshua Harlow
- Priority:
- Medium
- Drafter:
- Ivan Melnikov
- Direction:
- Approved
- Assignee:
- None
- Definition:
- Approved
- Series goal:
- None
- Implementation:
-
Unknown
- Milestone target:
- None
- Started by
- Completed by
Related branches
Related bugs
Sprints
Whiteboard
Possible backends include:
- kombu (convert current code to be a backend)
- oslo.messaging (really, we should support it)
- http/rest
- websockets? engine polling for result? workers polling for tasks?
- zookeeper
- we love it =)
- re-use some of jobboard code?
I (iv_m) think that these transport can operate on higher level of abstraction (e.g. know about tasks and flows and everything), so transport implemented this way might be simpler and generally better for our tasks than e.g. generic kombu virtual transport or oslo.messaging backend using same technologies.
Another option would be to switch to oslo.messaging completely and (possibly) implement some of backends as oslo.messaging backends.
Work Items
Dependency tree
* Blueprints in grey have been implemented.
