Support a process executor

Registered by Joshua Harlow

It appears like we can either use the existing executor that can work with processes in taskflow (or find out why it doesn't work) or even potentially use the more advanced process control framework that comes with a project like https://pypi.python.org/pypi/billiard which is internally used by celery (so it might be decent to use?). Then we can not only execute in a single thread (serial) and in multithreads (threaded or greenlet/eventlet threaded) using the executor model or in a distributed manner (when that arrives) but also using processes (possible also using the executor model?).

Might be neat to try and if it will work and if not see whats missing/broken.

Blueprint information

Status:
Complete
Approver:
Joshua Harlow
Priority:
Low
Drafter:
Joshua Harlow
Direction:
Approved
Assignee:
Joshua Harlow
Definition:
Approved
Series goal:
None
Implementation:
Implemented
Milestone target:
None
Started by
Joshua Harlow
Completed by
Joshua Harlow

Related branches

Sprints

Whiteboard

Another simple library that uses a similiar approach (and might be easier to use than billiard):
  - https://github.com/spotify/luigi/blob/master/luigi/worker.py

Gerrit topic: https://review.openstack.org/#q,topic:bp/process-executor,n,z

Addressed by: https://review.openstack.org/121280
    Work on getting a process executor working (WIP)

Addressed by: https://review.openstack.org/127114
    Rework pieces of the task listener/watcher capability

Addressed by: https://review.openstack.org/132376
    Ensure that failures can be pickled

Addressed by: https://review.openstack.org/132377
    Allow stopwatches to be restarted

Addressed by: https://review.openstack.org/132378
    Have tasks be able to provide copy() methods

Addressed by: https://review.openstack.org/132380
    Stop returning atoms from execute/revert methods

Addressed by: https://review.openstack.org/134690
    Get event/notification sending working correctly

Addressed by: https://review.openstack.org/129527
    Rework pieces of the task callback capability

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.