Create Scheduler Python Library

Registered by John Garbutt on 2014-02-25

In an effort to split out the nova-scheduler into its own project (gantt), this takes the first step on introducing a clear split between nova-scheduler and the rest of nova.

This is just refactoring.

A later blueprint will look at enforcing the split, and ensuring things like DB access in the scheduler are limited to a single DB table, and ensures the rest of Nova never populates that table directly (only via the scheduler lib), and other such important details.

Blueprint information

Status:
Complete
Approver:
John Garbutt
Priority:
High
Drafter:
John Garbutt
Direction:
Approved
Assignee:
Sylvain Bauza
Definition:
Approved
Series goal:
Accepted for juno
Implementation:
Implemented
Milestone target:
milestone icon 2014.2
Started by
John Garbutt on 2014-04-28
Completed by
John Garbutt on 2014-09-01

Related branches

Sprints

Whiteboard

Ready for discussion at the summit.

--
This assume( the library knows where to publish stats, ie. Gantt DB
This library should also propose some way to prepare reading config files by specifying which file to read for config. IMHO, it should be viewed as a GanttClient object, easing later the move to REST.

Something like:
from gantt.scheduler import library
ganttclient = library.GanttClient(conf=<conf>,*args, **kwargs)
ganttclient.publish_service_stats(ctx, service_name, service_stats)
 -sbauza

--

The idea hear is to create an API can be replaced by the GanttClient. So we could have a config that adds the "SchedulerAPI" class to load. By default it loads the nova one, but eventually, it can use the gnatt one.

The API right now, will just be nova code, doing what it does today, calling the DB, or calling the RPCAPI, its all about baby steps right now. Lets create the seams, see how they look, and see that the next steps are

--johnthetubaguy (26th Feb 2014)

Can you describe more the publish_service_stats() needed interface for Nova ? When looking at the resource_track, all code for updating stats is made thru the _update(self, context, values) classmethod which itself calls the conductor API for persisting the update.
Please correct me if I'm wrong, but instead of calling the conductor API, _update() should place a call to Gantt's publish_service_stats(). If so, we need to make sure there is a correct matching with attributes.
 - sbauza (7th March 2014)

Hi I am interested to contribute here, can you guys give me something so that I can start looking into it. I am asking here specifically because I dont know about the progress of this blueprint.
--Digambar (11th Mar 2014)

I am interested to contribute here too. --leandro-i-costantino (11th March 2014)

---
I'll put a draft proposal by the next 2 weeks so leandro and digambar you could review it (sbauza, 11t March 2014)
---
You can find below a draft review for the first implementation. As Nova is currently under FF, let's take the opportunity to review the code before Juno starts so we could push it early. (sbauza, 13th March 2014)

Only n0ano, johnthetubaguy, leandro and digambar can currently review it, so please let me know if you're interested in reviewing so I could put you in the list
---

Added a possible set of steps to implement this --johnthetubaguy (18th March 2014)

Removed from next, as next is now reserved for near misses from the last milestone --johnthetubaguyReady for discussion at the summit.

--
This assumes the library knows where to publish stats, ie. Gantt DB
This library should also propose some way to prepare reading config files by specifying which file to read for config. IMHO, it should be viewed as a GanttClient object, easing later the move to REST.

Something like:
from gantt.scheduler import library
ganttclient = library.GanttClient(conf=<conf>,*args, **kwargs)
ganttclient.publish_service_stats(ctx, service_name, service_stats)
 -sbauza

--

The idea hear is to create an API can be replaced by the GanttClient. So we could have a config that adds the "SchedulerAPI" class to load. By default it loads the nova one, but eventually, it can use the gnatt one.

The API right now, will just be nova code, doing what it does today, calling the DB, or calling the RPCAPI, its all about baby steps right now. Lets create the seams, see how they look, and see that the next steps are

--johnthetubaguy (26th Feb 2014)

Can you describe more the publish_service_stats() needed interface for Nova ? When looking at the resource_track, all code for updating stats is made thru the _update(self, context, values) classmethod which itself calls the conductor API for persisting the update.
Please correct me if I'm wrong, but instead of calling the conductor API, _update() should place a call to Gantt's publish_service_stats(). If so, we need to make sure there is a correct matching with attributes.
 - sbauza (7th March 2014)

Hi I am interested to contribute here, can you guys give me something so that I can start looking into it. I am asking here specifically because I dont know about the progress of this blueprint.
--Digambar (11th Mar 2014)

I am interested to contribute here too. --leandro-i-costantino (11th March 2014)

---
I'll put a draft proposal by the next 2 weeks so leandro and digambar you could review it (sbauza, 11t March 2014)
---
You can find below a draft review for the first implementation. As Nova is currently under FF, let's take the opportunity to review the code before Juno starts so we could push it early. (sbauza, 13th March 2014)

Only n0ano, johnthetubaguy, leandro and digambar can currently review it, so please let me know if you're interested in reviewing so I could put you in the list
---

Added a possible set of steps to implement this --johnthetubaguy (18th March 2014)

Removed from next, as next is now reserved for near misses from the last milestone --johnthetubaguy

Marking this blueprint as definition: Drafting. If you are still working on this, please re-submit via nova-specs. If not, please mark as obsolete, and add a quick comment to describe why. --johnthetubaguy (20th March 2014)

--
Will issue a nova-specs review by tomorrow -sbauza (20th Mar 2014)

--
Gerrit topic: https://review.openstack.org/#q,topic:bp/Scheduler,n,z

Addressed by: https://review.openstack.org/82133
    Propose Scheduler library blueprint

Given previous conversations of intent, and code already partially ready, targeting for Juno-1 --johnthetubaguy (28th April 2014)

Addressed by: https://review.openstack.org/82778
    Create a Scheduler client library

I think the TODO list is out of wack with the final spec, so it makes this ready to review? --johnthetubaguy 30th May

Addressed by: https://review.openstack.org/97232
    Move scheduler-client logic to Scheduler Manager

Addressed by: https://review.openstack.org/97837
    Add pci_stats and host_ip to ComputeNode object fields

Addressed by: https://review.openstack.org/79294
    Add host_ip to compute node object

Reducing priority because this seems unlikely to make juno-1, also added missing dependency, given the current patches up for review --johnthetubaguy 9th June 2014

This did not get reviewed and approved in time :( moving to juno-2 --johnthetubaguy 10 June 2014

Gerrit topic: https://review.openstack.org/#q,topic:bp/isolate-scheduler-db,n,z

Addressed by: https://review.openstack.org/101858
    Create a Scheduler client library

Gerrit topic: https://review.openstack.org/#q,topic:bp/scheduler-lib-patch26,n,z

Gerrit topic: https://review.openstack.org/#q,topic:bug/1333498,n,z

Gerrit topic: https://review.openstack.org/#q,topic:bp/bp,n,z

Addressed by: https://review.openstack.org/104556
    Add support for select_destination in Scheduler client

Addressed by: https://review.openstack.org/105747
    POC - Scheduler client for ResourceTracker

Reviews do not have +2s and are not close to approval, so moving to juno-3, but please move this back to juno-2, should you get your patches approved in time. --johnthetubaguy

Outstanding patches:

Patches to review
==============

Gerrit topic: https://review.openstack.org/#q,topic:bp/scheduler-lib,n,z

https://review.openstack.org/#/c/82778

https://review.openstack.org/#/c/104556

(?)

Work Items

Work items:
Create scheduler lib "seam" in nova (select_destination, update_node_stats): INPROGRESS

Dependency tree

* Blueprints in grey have been implemented.