Improving xz usage for developers & disk space constrained repositories (-dbgsym)

Registered by Dimitri John Ledkov on 2010-04-03

Fedora is doing it. Will reduce archive size tremendously. Will help Debian to consider the switch for squeeze+1. Dpkg supports it already in 12.04LTS.

Blueprint information

Status:
Not started
Approver:
Steve Langasek
Priority:
Low
Drafter:
Dimitri John Ledkov
Direction:
Needs approval
Assignee:
None
Definition:
Pending Approval
Series goal:
Accepted for raring
Implementation:
Unknown
Milestone target:
None

Whiteboard

User Stories:
Developers want to shorten the time it takes to build the packages locally, hence they need a way to disable long running xz compression which some packages are starting to use.
dbgsym repository constantly grows in size. To save disk space we want to shrink it as much as possible

Risks:
Fiddling with build can produce/hide build-failures, so it should be used with caution.
If some users of dbgsym repositories, do not have recent enough dpkg, they will not be able to use the repository any more.

Release Notes:
None. This feature is not user visible.

Session Notes:
At first we started to discuss solutions to switch the default compression options, but quickly concluded that there is no silver bullet and that this should not be done ahead of Debian. One interesting metric that came up is total time it takes to download & install packages which is a function of size/bandwidth + unpack time.

- There is no complete specification for this one
- Initial comments are that this will not happen as default before Debian
- Memory usage at unpack time should be tested on lower-end machines (e.g. Notebooks)

+ Can be done for selected packages
+ Langpacks, OO.o will benefit the most
+ Will make mirror smaller

+ Debian did it for udebs
+ Debian had a presentation about xz at Debconf12

Outcomes of UDS meeting.
* Packaging tool could detect whether compression gains benefit and conditionally put {un,}compressed data/control tarballs in deb. Solve 2/3 of the compression-benefit complaints.
* Colin says Ubuntu should not lead this; Debian should.
* Mirrors do not care about bandwidth. This benefit is imaginary.
* Un-constrained installation-image size makes this less important for us.
* Chad says user's installation time is the most important factor. If (shorter download time + longer uncompression time) > current time, then this fails, and we should abort XZ plans.

(?)

Work Items

Work items:
[xnox] enquire about possibly unsupported tooling with XZ: POSTPONED
[xnox] dpkg-deb/debhelper should support conditional compression (RFC sent to debian-devel): INPROGRESS
[xnox] decide time=money ratio in test for whether XZ is a net loss for users: POSTPONED
[xnox] calculate above time ratios for all packages in the archive: POSTPONED
[xnox] verify/check whoopsie, apport, daisy, lp-retracers are all ok supporting xz compressed ddebs (pitti says they run precise & dpkg-deb -x): DONE
[xnox] compress ddebs by default with XZ: DONE