Shrink the image build pipeline

Registered by Colin Watson on 2011-10-20

When preparing milestone releases, we often need to turn around fixes quickly. The pipeline from a developer source upload to a full set of updated image builds on all architectures is currently somewhere in the region of nine hours. We would like to make this much quicker.

Blueprint information

Status:
Started
Approver:
Steve Langasek
Priority:
High
Drafter:
Colin Watson
Direction:
Approved
Assignee:
Colin Watson
Definition:
Approved
Series goal:
Accepted for precise
Implementation:
Good progress
Milestone target:
milestone icon ubuntu-12.04-beta-1
Started by
Colin Watson on 2011-11-09

Related branches

Sprints

Whiteboard

Work items:
[adconrad] Spread legacy livefs building infrastructure across more machines as a stop-gap: DONE
[adconrad] Migrate live filesystem building into Launchpad, so that it can be scheduled across a larger number of machines, which would particularly help ARM: POSTPONED
[adconrad] Write launchpad-buildd slave implementation for live-build jobs: POSTPONED
[adconrad] Write buildd-manager and LP API implementation for live-build jobs: POSTPONED
[adconrad] Test live-build/launchpad-buildd integration on dogfood: POSTPONED
[adconrad] Rollout live-build/launchpad-buildd integration into production: POSTPONED
[cjwatson] Reorganise germinate to expose a useful Python interface: DONE
[cjwatson] Convert Launchpad to use python-germinate, aiming to reduce cron.germinate's run-time from around 10 minutes to around 2 minutes: DONE
[cjwatson] Move germinate before apt-ftparchive in Launchpad: POSTPONED
[cjwatson] If possible, make the publisher run every 30 minutes rather than every hour: DONE
[cjwatson] Fix the CD image system's archive mirror to be less inefficient in its handling of symlinks, which should save in the region of 10 minutes per serialised image build: DONE
[cjwatson] Consolidate main/universe archive mirrors on the CD image system into one, reducing races between certain sets of parallel builds: DONE
[nick-moffitt] Move CD image building to the new SAN: DONE
[lamont] Switch powerpc live filesystem builds to a faster machine (RT#48569 filed): TODO
[cjwatson] Restore some timestamp logging in live filesystem builds (Debian #630089): DONE
[cjwatson] Switch cdimage mirroring to mirror directly from cocoplum: DONE
[cjwatson] Look at transfer speed between nusakan and cocoplum and make sure sane (measured at ~330Mbit/s over rsync, seems good enough to me): DONE
[cjwatson] At least vaguely start on the giant debian-cd merge: TODO
[adconrad] Write a master CD build script (after live-build/lp-buildd integration is done): POSTPONED

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.