Expiration policies for Executions

Registered by Guy Paz

Currently there is no mechanism that removes past execution and that might lead to huge amounts of data being stored on DB / disk.
One simple workflow with a cron-trigger (For example monitor and heal workflow) can create large amounts of data in the DB even more in production environment when we will have dozens - say the least - of cron-trigers and over a time.

Proposed change
===========
In the mistral.conf will be a ‘expiration policy’ that defines what is expired executions (for example for that last x days), what to do with expired executions and how often to check for expired executions.

Policies options:
==========

  Expired criteria:
    - Older than x minutes (phase 1)
      older_than int in minutes

  Cleanup action:
    - Delete executions (phase 1)
      Delete expired executions via API so all TaskExecution and ActionExecution will
      be removed too (all executions that update_at date are older_than)

Only final states (SUCCESS and ERROR) executions will remove (phase 1)
In case that there is a workbook that contain 2 wfs that one call the second so the child wf finished and expired (SUCCESS) , but the parent still running (RUNNING) we will not remove the 'child' since the executions graph are still running. We will remove the all exec-wfs together when the parent will be done and expired (SUCCESS/ERROR)

Configuration:
=========
There are two parameters that define the Expiration Policy task (pahse 1):
"Evaluation interval" - time interval between Call Expiration Policy (in minutes).
"Older than" - which execution expired (every execution that 'update_at' < 'older_than' is expired)

** Note ** that default values are Null/None so there will be no Expiration Policy running.

This is the mistral.conf:

[expiration_policy]

#
# From mistral.config
#

# How often will the executions be evaluated (in minutes). For example
# for value 120 the interval will be 2 hours (every 2 hours).
# (integer value)
evaluation_interval = 5

# Evaluate from which time remove executions in minutes. For example
# when older_than = 60, remove all executions that finished a 60
# minutes ago or more. Note that only final state execution will
# remove ( SUCCESS / ERROR ) (integer value)
older_than = 10

Blueprint information

Status:
Complete
Approver:
None
Priority:
Medium
Drafter:
Guy Paz
Direction:
Needs approval
Assignee:
Guy Paz
Definition:
Approved
Series goal:
Accepted for liberty
Implementation:
Implemented
Milestone target:
milestone icon 1.0.0
Started by
Nikolay Makhotkin
Completed by
Renat Akhmerov

Related branches

Sprints

Whiteboard

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.