Asynchronous Processing in Glance (partial)

Registered by Mark Washenberger on 2013-04-25

Provide and use an internal library api for asynchronous processing in glance, to do things like image imports (asynchronously processing a copy-from). This can later be used to support image conversion and metadata extraction, though those should fall under other blueprints. A simple eventlet thread-based implementation may be provided, alongside something more like a remote rpc queue worker. There should be room for the implementation of the library api to be backed by other cool drivers, like ZeroVM (http://zerovm.org/wiki/The_Cloud_Hypervisor)

Blueprint information

Status:
Complete
Approver:
None
Priority:
High
Drafter:
None
Direction:
Approved
Assignee:
Nikhil Komawar
Definition:
Approved
Series goal:
Accepted for juno
Implementation:
Implemented
Milestone target:
milestone icon 2014.2
Started by
Mark Washenberger on 2013-06-18
Completed by
Thierry Carrez on 2014-09-09

Related branches

Sprints

Whiteboard

Targeting to h2 based on how centrally it figures in other blueprints

discussion items:-

requirements at :- https://etherpad.openstack.org/havana-glance-requirements

Gerrit topic: https://review.openstack.org/#q,topic:bp/async-glance-workers,n,z

Addressed by: https://review.openstack.org/42817
    Enable tasks data model and table for async worker

Addressed by: https://review.openstack.org/53935
    Adds tasks to db api

Addressed by: https://review.openstack.org/54198
    Adds domain level support for tasks

Addressed by: https://review.openstack.org/54252
    Introduce Task Info Table

Addressed by: https://review.openstack.org/54380
Add support for PartialTask list

Addressed by: https://review.openstack.org/59814
    Enable tasks REST API for async worker

Addressed by: https://review.openstack.org/44355
    Adds executor interface and example import script

Addressed by: https://review.openstack.org/58652
    Documentation for Glance tasks

How much work is left here? Should this be moved out from i-3?
-- flaper87 2014-02-18
(Response provided on IRC chat - attaching the same herewith temporarily for tracking purposes)
11:13:04 [ nikhil__] flaper87: hi there
11:14:25 [ flaper87] nikhil__: hey
11:16:28 [ nikhil__] flaper87: I'm about to send out a plan to markwash for the async bp
11:16:50 [ nikhil__] we've a consistently working exeutor here at rackspace with import and export use cases
11:17:07 [ nikhil__] which can be used to build a nice one in short period of time
11:17:27 [ nikhil__] also, can share the details about the plan with you if interested
11:17:37 [ flaper87] nikhil__: awesome, yeah please. I am
11:17:39 [ nikhil__] (all this is in reference to you comment on the BP)
11:17:46 [ nikhil__] ohk sure
11:18:00 [ nikhil__] think we can definitely get this in i-3
11:18:17 [ nikhil__] on regular deadlines
11:18:32 [ nikhil__] some work is started at https://review.openstack.org/#/c/44355/
11:19:18 * flaper87 clicks
11:19:52 [ flaper87] nikhil__: starred, I'll make sure I review that
11:20:09 [ flaper87] so, you think it is doable for i-3
11:20:10 [ flaper87] that's awesome
11:20:24 [ flaper87] I saw many other bps linked to the async-workers one
11:21:54 [ nikhil__] flaper87: yeah, based on discussion with markwash last week we planned async bp and import for i-3
11:22:07 [ flaper87] nikhil__: awesome!
11:22:19 [ nikhil__] and export to a little possibility
11:22:22 [ nikhil__] cool

This work would take a lot of input from this paste http://paste.openstack.org/show/66940/ (which is Rackspace's internal proof of concept implementation as a placeholder for some work done on this BP) as well as from review feedback and suggestions provided by the community.

markwash untagged 2014-03-03

Addressed by: https://review.openstack.org/75179
    Ensures that task.message is of type unicode

Addressed by: https://review.openstack.org/#/c/78050
    Return 405 when attempting DELETE on /tasks

Addressed by: https://review.openstack.org/76445
    Adds TaskStub class

Addressed by: https://review.openstack.org/83257
    Get rid of TaskDetails in favor of TaskStub

Addressed by: https://review.openstack.org/85211
    Basic Eventlet Task Executor with taskflow

Latest discussion and updates from the Atlanta summit at https://etherpad.openstack.org/p/juno-tasks-review-and-taskflow

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.