Integrate CBuild/LAVA scheduler

Registered by Paul Sokolovsky

Now that we can do CBuild builds in LAVA on the lowest level, we need to update CBuild scheduler to submit jobs to LAVA for execution (instead of using its own adhoc build dispatcher).

Blueprint information

Status:
Complete
Approver:
Данило Шеган
Priority:
High
Drafter:
Milo Casagrande
Direction:
Approved
Assignee:
Stevan Radaković
Definition:
Approved
Series goal:
Accepted for trunk
Implementation:
Implemented
Milestone target:
milestone icon 2013.01
Started by
Paul Sokolovsky
Completed by
Stevan Radaković

Related branches

Sprints

Whiteboard

Meta:
Headline: CBuild updated to support scheduling build via LAVA
Acceptance: CBuild web frontend and backend can schedule and do GCC builds on LAVA as the build farm.

This is follow up to https://blueprints.launchpad.net/linaro-infrastructure-misc/+spec/toolchain-build-in-lava
Driven by the http://cards.linaro.org/browse/CARD-195

[pfalcon 2012-11-27] Started discussion with LAVA people of means to publish big build artifacts from LAVA
[pfalcon 2012-11-29] Bootstrap meeting with Stevan
[pfalcon 2012-12-05] Successful mock gcc build in LAVA prod: http://validation.linaro.org/lava-server/scheduler/job/40969 . Instructions: https://wiki.linaro.org/Platform/Infrastructure/CBuildLava#Scheduling_a_mock_gcc_build
[pfalcon 2012-12-05] CBuild server bootstrap instructions from Michael: http://bazaar.launchpad.net/~linaro-toolchain-dev/cbuild/cbuild-init/view/head:/README
[pfalcon 2012-12-05] Filed https://bugs.launchpad.net/lava-dispatcher/+bug/1086753 for adding publishing support to LAVA (LAVA people invite to submit patches)
[stevanr 2012-12-06] Local installation done on ec2 instance http://ec2-54-243-13-36.compute-1.amazonaws.com/helpers/ . It still has some problems, currently communicating with michaelh about those.
[pfalcon 2012-12-07] Initial impl for integrating manual spawn with LAVA pushed to https://code.launchpad.net/~linaro-infrastructure/tcwg-web/cbuild-lava
[pfalcon 2012-12-10] Having issues with redirects on EC2 sanbox, Stevan is on it
[pfalcon 2012-12-10] Deployed current changes for lava integration ("manual spawn" and "bump" support), it's possible to launch a job into LAVA on EC2 sandbox now
[stevanr 2012-12-12] New ec2 CBuild instance ready at http://ec2-23-22-175-203.compute-1.amazonaws.com/helpers/
[pfalcon 2012-12-13] Common internal CBuild API for scheduling jobs (abstracting LAVA support, etc): common.schedule_job() method, exported as script with schedulejob.py (to be called by command-line tools, like cbuild-tools).
[danilo 2012-12-28] We didn't manage to finish even the polling and buildlog functionality, thus we are still not done even with an extra week after 2012.12 was due. Moving to next cycle, and assigning to Stevan (pairing him with Milo).

(?)

Work Items

Work items:
[pfalcon] Set up mock gcc build in LAVA for quick test turnaround: DONE
[pfalcon] Discuss ways to publish big build artifacts from LAVA: DONE
Publish results of LAVA builds: POSTPONED
[pfalcon] Attach build logs to LAVA result bundles: DONE
[stevanr] Setup local devel environment for CBuild master: DONE
[stevanr] Setup EC2 sandbox for CBuild master: DONE
[stevanr] Prepare easy-to use scripts to deploy EC2 sandbox for CBuild: DONE
[pfalcon] Add basic LAVA integration for CBuild frontend (spawn, bump actions): DONE
[stevanr] Refactor .lock file handling (merge .lava-id data into it, without affecting native cbuild jobs): DONE
[stevanr] Implement cronjob to poll and update LAVA job status: DONE
[stevanr] Pull build logs from LAVA on job completion: DONE
[pfalcon] Add LAVA support for CBuild automated builds (cbuild-tools): DONE
[pfalcon] Introduce common CBuild API to schedule jobs: DONE
[pfalcon] Document CBuild/LAVA integration architecture and usage: DONE

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.