[data processing] Changes to streamline cluster creation

Registered by Chad Roberts

** Most likely a Kilo item **
The current workflow for cluster creation is often sub-optimal for the most common use cases. This blueprint aims to streamline the process for common cluster creation use cases.

For instance, common clusters have 1 master and N workers, for a given plugin the master always has the same set of processes and so does the worker.
Given that, ideally, we could provide cluster launch workflow w/ just a "how big (how many workers)?" question. This might require either the sahara service or the plugins to provide some sort of "default master/worker" template, or possibly the dashboard could have defaults generated (but wouldn't be useable outside of the dashboard, so I prefer the default templates being defined elsewhere)

Even if we don't get the "default templates", the UI cluster launch workflow could have a new tab that would allow the user to modify the number of instances of each node group. When that is submitted, the dashboard could modify the cluster creation call to use the updated instance numbers (which might require the dashboard to "reconstruct" the template on the fly).

In either case, the spirit of this blueprint is that the user should NOT have to recreate a whole cluster template just to change the number of workers in their launched cluster.

Blueprint information

Status:
Complete
Approver:
David Lyle
Priority:
Medium
Drafter:
Chad Roberts
Direction:
Needs approval
Assignee:
None
Definition:
Obsolete
Series goal:
None
Implementation:
Unknown
Milestone target:
None
Completed by
David Lyle

Related branches

Sprints

Whiteboard

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.