Redesign Mistral Scheduler
Based on the etherpad https:/
...
It uses too agressive polling, it creates significant load on DB
Due to agressive DB polling in Scheduler Mistral can't be scaled well. Every new engine node increases load on DB
There are situations when a delayed call (the entity that the Scheduler works with) is deleted from DB but, in fact, is not processed yet. It leaves the system in an inconsistent state (a WF may get stuck in RUNNING state)
...
Blueprint information
- Status:
- Complete
- Approver:
- Dougal Matthews
- Priority:
- High
- Drafter:
- Renat Akhmerov
- Direction:
- Approved
- Assignee:
- Renat Akhmerov
- Definition:
- New
- Series goal:
- Accepted for stein
- Implementation:
-
Implemented
- Milestone target:
-
train-1
- Started by
- Renat Akhmerov
- Completed by
- Renat Akhmerov
Related branches
Related bugs
Sprints
Whiteboard
Gerrit topic: https:/
Addressed by: https:/
WIP: designing a new scheduler
Gerrit topic: https:/
Addressed by: https:/
WIP: Integrating the new scheduler into the system
Addressed by: https:/
Add a migration to create the scheduled_jobs table
Addressed by: https:/
Add db api tests for scheduled jobs
Addressed by: https:/
Allow to delete multiple objects with advanced filters
Addressed by: https:/
WIP: Bulk update and delete of stored scheduled jobs
Addressed by: https:/
WIP: Improve new scheduler
