[data processing] Changes to streamline cluster creation
** Most likely a Kilo item **
The current workflow for cluster creation is often sub-optimal for the most common use cases. This blueprint aims to streamline the process for common cluster creation use cases.
For instance, common clusters have 1 master and N workers, for a given plugin the master always has the same set of processes and so does the worker.
Given that, ideally, we could provide cluster launch workflow w/ just a "how big (how many workers)?" question. This might require either the sahara service or the plugins to provide some sort of "default master/worker" template, or possibly the dashboard could have defaults generated (but wouldn't be useable outside of the dashboard, so I prefer the default templates being defined elsewhere)
Even if we don't get the "default templates", the UI cluster launch workflow could have a new tab that would allow the user to modify the number of instances of each node group. When that is submitted, the dashboard could modify the cluster creation call to use the updated instance numbers (which might require the dashboard to "reconstruct" the template on the fly).
In either case, the spirit of this blueprint is that the user should NOT have to recreate a whole cluster template just to change the number of workers in their launched cluster.
Blueprint information
- Status:
- Complete
- Approver:
- David Lyle
- Priority:
- Medium
- Drafter:
- Chad Roberts
- Direction:
- Needs approval
- Assignee:
- None
- Definition:
- Obsolete
- Series goal:
- None
- Implementation:
- Unknown
- Milestone target:
- None
- Started by
- Completed by
- David Lyle