Deploy Celery support for LAVA server
Celery is an asynchronous task queue/job queue based on distributed message passing.
LAVA Celery makes possible to use Celery easily from LAVA without extra configuration.
The deployment of LAVA Celery will allow to offload the tasks to workers.
In addition, it will resolve the IO bottleneck when several parallel media are
created with l-m-c/l-a-m-c and will get rid of the current workaround (serialized builds).
Blueprint information
- Status:
- Complete
- Approver:
- Paul Larson
- Priority:
- Essential
- Drafter:
- Paul Larson
- Direction:
- Approved
- Assignee:
- Zygmunt Krynicki
- Definition:
- Approved
- Series goal:
- Accepted for trunk
- Implementation:
-
Implemented
- Milestone target:
-
2012.03
- Started by
- Fathi Boudra
- Completed by
- Zygmunt Krynicki
Related branches
Related bugs
Sprints
Whiteboard
[zkrynicki, 2011-10-26] should celery be deployed NOW or can we wait until after we do the non-dpkg deployment?
[fboudra, 2011-10-26] nothing should be done on this blueprint before it's specified (WI) and approved.
[zkrynicki, 2011-10-26] I was asking about the order of the actual work, shoud we make sure celery can be deployed before we change the deployment method (flexible lava deployment BP)
[zkrynicki, 2011-11-28] re-target to 2011.12.
[fboudra, 2011-12-22] re-target to 2012.01.
[fboudra, 2012-01-26] re-target to 2012.02.
[fboudra, 2012-02-01] add rabbitmq setup WI. It is blocked until the cloud has been set up, which is blocked until the new e-net switch arrives.
[fboudra, 2012-02-24] re-target to 2012.03 and set priority to essential. The cloud has been set up.
[zkrynicki, 2012-03-22] Setting implementation to deployment
[zkrynicki, 2012-05-18] postponing celery startup as it is not relevant anymore, marking as implemented
Meta:
Headline: LAVA can do complex and long computations in the background if necessary
Acceptance: LAVA celery is installed by default and Validation lab LAVA installation is ready for celery
Work Items
Work items:
Create lava-celery: DONE
Add lava-celery to default requirement.txt file in lp:lava-deployment-tool: DONE
Create upstart job for celeryd: DONE
Create upstart job for celerybeat: DONE
Create upstart job for celerycam: DONE
Make vhost configurable: DONE
Fix upstart scripts with regard to tracking the right pid: POSTPONED
Create different rabbitmq vhosts for production instance: DONE
Allow more than one queue to be used: TODO
Configure a single worker for celery in the validation lab server (staging, done automatically): DONE
Create a simple task (lava-vcs) that can be started to ensure rabbitmq and celery work correctly: DONE
Dependency tree
* Blueprints in grey have been implemented.
