LAVA Dispatcher

LAVA Dispatcher Cloudified

Registered by Zygmunt Krynicki

To use additional computing resources available in the lab, we need to allow LAVA to run parts of the stack in the cloud.

For architectural simplicity, we should only change the dispatcher so that it can offload the heavy parts of the work to other machines.

The dispatcher now will optionally support rabbitmq (or other broker) and celery to coordinate and distribute the workload.

Blueprint information

Status:
Complete
Approver:
Paul Larson
Priority:
Essential
Drafter:
Zygmunt Krynicki
Direction:
Approved
Assignee:
Zygmunt Krynicki
Definition:
Approved
Series goal:
None
Implementation:
Implemented
Milestone target:
milestone icon 2012.03
Started by
Zygmunt Krynicki
Completed by
Zygmunt Krynicki

Sprints

Whiteboard

[zkrynicki, 2012-03-15] In general this seems to work fine. I need to polish it a bit and arrange for proper stdout/stderr capture.
[zkrynicki, 2012-03-22] Configuration will be done next cycle as we see how deployment works in practice. For now it works as advertised with the added exception that the node that schedules remote commands must run rabbitmq (there is no way to specify alternate host than localhost). This works fine for us in validation.linaro.org. Once we figure out a sane way to handle settings across tools and server extensions we can fix this (since lava-celery already pulls in configuration from lava-server, simply the tool part does not). Setting implementation to deployment. The code is there, let's use it
[zkrynicki, 2012-05-18] Marking as implemented as there is nothing more to do and I want to get this out of my list

Meta:
Headline: The LAVA Dispatcher can now be distributed to other machines using rabbitmq and celery
Acceptance:
1. Dispatcher runs as a celery task
2. system is configured to allow dispatcher jobs on at least one system
3. exit code and output stream are not lost by having it on another machine
Roadmap id: LAVA2012-FAST-MODELS-IN-LAVA

(?)

Work Items

Work items:
Add support for setting up celery and kombu: DONE
Arrange the code so that we can have a compatibility shim and the real work separated: DONE
[mwhudson] Test on panda: DONE
Add support for redirecting stdout to kombu exchange: DONE
Add support for waiting for a celery task to finish: DONE
Add support for streaming output from running task in the shim: DONE
Ensure that real return code is propagated to the shim: DONE
Manage sentry configuration via lava-server if available: POSTPONED
Manage sentry configuration via command line if lava-server is not available: POSTPONED
Turn lava-dispatcher into lava-tool: DONE
Document how to set up the server (quickstart/howto): POSTPONED

This blueprint contains Public information 
Everyone can see this information.