[EDP] Job Manager Components

Registered by Alexander Kuznetsov

This component is responsible for Job management lifecycles: creating, scheduling, monitoring etc.

We assume to support following job types:
  * Pig
  * Hive
  * Jar file
  * Oozie workflow (not in version 0.3)
  * YARN application (not in version 0.3)
  * Streaming api scripts (not in version 0.3)

Object model for this component has two items: job and job execution.

Job Object
  * Name
  * Description
  * Job type
  * Mains (location of main job binary)
  * Libs (locations of supporting job binaries)

Job Execution
  * Job Id
  * Input Data Source Id
  * Output Data Source Id
  * Start time
  * End time
  * Cluster Id
  * Info (job status from Oozie)
  * Progress
  * Oozie Job Id
  * Return code
  * Job Configs
  * Logs (Optional, possible not will be in version 0.3)

This component should implement following methods:
  * Create job
  * List jobs
  * Delete job
  * Execute job. This method retrieves the job binaries listed in the Job object referenced by Job Id, applies user specified configuration for the job, and starts the job on an already running cluster.
  * Terminate job instance
  * List job instances
  * Get job instance status

Blueprint information

Status:
Complete
Approver:
Sergey Lukjanov
Priority:
High
Drafter:
Trevor McKay
Direction:
Approved
Assignee:
Nadya Privalova
Definition:
Approved
Series goal:
Accepted for 0.3
Implementation:
Implemented
Milestone target:
milestone icon 0.3
Started by
Nadya Privalova
Completed by
Sergey Lukjanov

Related branches

Sprints

Whiteboard

https://etherpad.openstack.org/edp_v3_components

Gerrit topic: https://review.openstack.org/#q,topic:bp/edp-data-discovery-component,n,z

Addressed by: https://review.openstack.org/38627
    Added REST API for job and data source with simple validation

Addressed by: https://review.openstack.org/42304
    First version of job manager

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.