Shrink the image build pipeline

Registered by Colin Watson

When preparing milestone releases, we often need to turn around fixes quickly. The pipeline from a developer source upload to a full set of updated image builds on all architectures is currently somewhere in the region of nine hours. We would like to make this much quicker.

Blueprint information

Steve Langasek
Colin Watson
Colin Watson
Series goal:
Accepted for precise
Good progress
Milestone target:
milestone icon ubuntu-12.04-beta-1
Started by
Colin Watson

Related branches



Work items:
[adconrad] Spread legacy livefs building infrastructure across more machines as a stop-gap: DONE
[adconrad] Migrate live filesystem building into Launchpad, so that it can be scheduled across a larger number of machines, which would particularly help ARM: POSTPONED
[adconrad] Write launchpad-buildd slave implementation for live-build jobs: POSTPONED
[adconrad] Write buildd-manager and LP API implementation for live-build jobs: POSTPONED
[adconrad] Test live-build/launchpad-buildd integration on dogfood: POSTPONED
[adconrad] Rollout live-build/launchpad-buildd integration into production: POSTPONED
[cjwatson] Reorganise germinate to expose a useful Python interface: DONE
[cjwatson] Convert Launchpad to use python-germinate, aiming to reduce cron.germinate's run-time from around 10 minutes to around 2 minutes: DONE
[cjwatson] Move germinate before apt-ftparchive in Launchpad: POSTPONED
[cjwatson] If possible, make the publisher run every 30 minutes rather than every hour: DONE
[cjwatson] Fix the CD image system's archive mirror to be less inefficient in its handling of symlinks, which should save in the region of 10 minutes per serialised image build: DONE
[cjwatson] Consolidate main/universe archive mirrors on the CD image system into one, reducing races between certain sets of parallel builds: DONE
[nick-moffitt] Move CD image building to the new SAN: DONE
[lamont] Switch powerpc live filesystem builds to a faster machine (RT#48569 filed): TODO
[cjwatson] Restore some timestamp logging in live filesystem builds (Debian #630089): DONE
[cjwatson] Switch cdimage mirroring to mirror directly from cocoplum: DONE
[cjwatson] Look at transfer speed between nusakan and cocoplum and make sure sane (measured at ~330Mbit/s over rsync, seems good enough to me): DONE
[cjwatson] At least vaguely start on the giant debian-cd merge: TODO
[adconrad] Write a master CD build script (after live-build/lp-buildd integration is done): POSTPONED


Work Items

Dependency tree

* Blueprints in grey have been implemented.