Asynchronously process tasks using celery
Some tasks contain enough actions and work that they take a long time to respond and may time out before returning. This won't affect the task itself, since it should still complete before attempting to return, but the user may be confused by this and while this task is processing an API worker is blocked.
The majority of tasks could be processed asynchronously, but this would mean that tasks don't respond at all beyond basic validation and you must query for their status. This could be considered an API breaking change and require an introduction of a v2 API.
The work here would be to move task processing to celery workers. This logic is currently handled in the TaskView class and could easily enough be moved to a celery worker construct. We can then tag certain tasks as async or not, and for existing tasks we can effectively make it so that the Views wait until a worker completes before they respond.
Ideally we should split the API and task processing, so all tasks are handled asynchronously. A task is started, and will respond with a 202. User can then query for status, although Adjutant will send emails and such as the task gets processed. We could also introduce notification support of some sort for task status.
Work items vaguely include:
- move task processing to celery workers
- open task list and task query APIs to users with sensible permissions and filtering for non-admins
- create notifications for tasks as they are processed
Related to this work is may be the redefinition of taskviews and the current way we use them (for more than outright task processing).
Blueprint information
- Status:
- Not started
- Approver:
- None
- Priority:
- Undefined
- Drafter:
- Adrian Turjak
- Direction:
- Needs approval
- Assignee:
- None
- Definition:
- New
- Series goal:
- None
- Implementation:
-
Unknown
- Milestone target:
- None
- Started by
- Completed by