Remove DB between scheduler and compute nodes

Registered by Boris Pavlovic on 2013-07-23

Current way how work Nova.scheduler is not optimal there is the way to make it more scalable & flexible.

New scheduler design doc:

More details about limitations of current scheduler you can see here:

There is a community effort to make 1 scheduler lib or 1 scheduler as a Service, to reduce amount of duplication of code.

During the H cycle we tried to move all common parts of scheduler to oslo.scheduler:

But we faced a lot of problems. The main problems was, that host_states are deeply connected with db.models that represent host states. And scheduler use directly projects db.

The second thing that we faced was the bad performance of scheduler. The main reason is the we are storing & updating all data in DB + on each request we are fetching all data from DB.

So as a solution we try to resolve these 2 problems:
1) Each scheduler should store all host_states locally (in some object)
2) There should be mechanism that sync all host_states in different backends
(using memcahced or sqla backends, to store updates)
This is required because another way are fanouts RPC calls, that doesn't scale..
3) There should be namespaces in host_states (to store in the same time data from different services) and be able to use filters from different projects in the same time.
4) host_state is just JSON like object.
5) Scheduler should have RPC calls to update/get host states + nova.manger, nova.api should call them instead calling nova.db.api

Our goal is to make queue of next patches:
1) Add new db.models to implement backend of synchronizer
2) Add synchronizer
3) Switch Scheduler to use synchronizer + add new RPC calls to scheduler + call from compute.manager it
4) Implement in Scheduler all required RPC calls to get/list host states
5) Remove compute_nodes table, and use scheduler in db.api as implementation
6) Remove comupte_* methods from db.api and use directly scheduler

This is the first that will make Scheduler faster, flexible & much more independent from Nova code.
Which will make unification of scheduler (or making scheduler aaS much simpler)

Blueprint information

Russell Bryant
Boris Pavlovic
Needs approval
Boris Pavlovic
Series goal:
Needs Code Review
Milestone target:
milestone icon next
Started by
Boris Pavlovic on 2013-12-03

Related branches



I see there are two things the scheduler does...
* compute nodes send updates (currently via the DB)
* scheduler picks what node is required
Basically this introduces using RPC and memcache instead of database calls to perform updates, plus syncing the data between schedulers.
I would like to see the memcache dependency as optional during Icehouse, with a view (if it works well) make it mandatory in J

Gerrit topic:,topic:bp/no-db-scheduler,n,z

Addressed by:
    Scheduler redesign for in-memory state storage.

Addressed by:
    A migration which creates tables for Synchronizer

I just spoke to Boris about this blueprint. i'd like the description to be updated to reduce the scope. This blueprint should not include work on a separate scheduler service. it should also not include anything related to other projects. I'd like to see it specifically limited to no-db work, and just for nova to start with. --russellb

Replaced description, to cover only minimal scope of changes -- boris-42

Addressed by:
    Decoupling scheduler from database access.

Addressed by:
    Switches to Synchronizer.

Addressed by:
    Memcached backend is added.

Its too late for merging this in icehouse, defer to juno. Also we need to re-asses dependencies here, I would like to see the scheduler split merge first, so this can be totally optional. --johnthetubaguy

Unapproved - please re-submit via nova-spec --johnthetubagy (20th March 2014)


Work Items