Proccessing data collections in Data Flow

Registered by Renat Akhmerov

Data Flow mechanism should be able to deal with collections, not only with fixed parameters. A typical scenario is when we want to run the same task multiple times with different sets of parameters. For instance, we want to create 100 VMs and it would be ugly to have 100 separate tasks in a workflow definition. So in this case we task could be smart enough to understand how to iterate through input collection containing parameter sets.

The current idea is as follows:

attach-vm:
    for-each:
        vminfo: $.vms
    action: create_vm name=$.vminfo.name flavor=$.vminfo.flavor
    action: create_vm name={$.server_name} image_id={$.vminfo.image_id} image_id={$.vminfo.flavor_id}

So 'for-each' just makes the task run over an array of elements where each element may be any structure (e.g. dictionary).

See https://etherpad.openstack.org/p/mistral-workflow-dataflow-collections for more details.

Blueprint information

Status:
Complete
Approver:
Renat Akhmerov
Priority:
High
Drafter:
Renat Akhmerov
Direction:
Approved
Assignee:
Nikolay Makhotkin
Definition:
New
Series goal:
Accepted for juno
Implementation:
Implemented
Milestone target:
milestone icon 2015.1
Started by
Renat Akhmerov
Completed by
Renat Akhmerov

Related branches

Sprints

Whiteboard

note: very similar :
A: start_server_1 (type == nova.server)
B: do something else
C: start_server_2 (type == nova.server)

How do we get different parameters into A and C via the context?

Gerrit topic: https://review.openstack.org/#q,topic:bp/mistral-dataflow-collections,n,z

Addressed by: https://review.openstack.org/128190
    Implement for-each task property

Addressed by: https://review.openstack.org/127928
    Add validating of 'for-each' task property DSL

Gerrit topic: https://review.openstack.org/#q,topic:for-each,n,z

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.