Deploy Celery support for LAVA server

Registered by Fathi Boudra on 2011-10-07

Celery is an asynchronous task queue/job queue based on distributed message passing.
LAVA Celery makes possible to use Celery easily from LAVA without extra configuration.

The deployment of LAVA Celery will allow to offload the tasks to workers.
In addition, it will resolve the IO bottleneck when several parallel media are
created with l-m-c/l-a-m-c and will get rid of the current workaround (serialized builds).

Blueprint information

Status:
Complete
Approver:
Paul Larson
Priority:
Essential
Drafter:
Paul Larson
Direction:
Approved
Assignee:
Zygmunt Krynicki
Definition:
Approved
Series goal:
Accepted for trunk
Implementation:
Implemented
Milestone target:
milestone icon 2012.03
Started by
Fathi Boudra on 2011-11-18
Completed by
Zygmunt Krynicki on 2012-05-18

Related branches

Sprints

Whiteboard

[zkrynicki, 2011-10-26] should celery be deployed NOW or can we wait until after we do the non-dpkg deployment?
[fboudra, 2011-10-26] nothing should be done on this blueprint before it's specified (WI) and approved.
[zkrynicki, 2011-10-26] I was asking about the order of the actual work, shoud we make sure celery can be deployed before we change the deployment method (flexible lava deployment BP)
[zkrynicki, 2011-11-28] re-target to 2011.12.
[fboudra, 2011-12-22] re-target to 2012.01.
[fboudra, 2012-01-26] re-target to 2012.02.
[fboudra, 2012-02-01] add rabbitmq setup WI. It is blocked until the cloud has been set up, which is blocked until the new e-net switch arrives.
[fboudra, 2012-02-24] re-target to 2012.03 and set priority to essential. The cloud has been set up.
[zkrynicki, 2012-03-22] Setting implementation to deployment
[zkrynicki, 2012-05-18] postponing celery startup as it is not relevant anymore, marking as implemented

Meta:
Headline: LAVA can do complex and long computations in the background if necessary
Acceptance: LAVA celery is installed by default and Validation lab LAVA installation is ready for celery

(?)

Work Items

Work items:
Create lava-celery: DONE
Add lava-celery to default requirement.txt file in lp:lava-deployment-tool: DONE
Create upstart job for celeryd: DONE
Create upstart job for celerybeat: DONE
Create upstart job for celerycam: DONE
Make vhost configurable: DONE
Fix upstart scripts with regard to tracking the right pid: POSTPONED
Create different rabbitmq vhosts for production instance: DONE
Allow more than one queue to be used: TODO
Configure a single worker for celery in the validation lab server (staging, done automatically): DONE
Create a simple task (lava-vcs) that can be started to ensure rabbitmq and celery work correctly: DONE

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.