apt-get hashsum/size mismatch because s3 mirrors don't support http pipelining correctly

Apt log of import

Steps to repoduce (reliably):

1) Launch an instance in EC2
2) Run: sed -i "s,ec2.archive.ubuntu.com,ec2.archive.ubuntu.com.s3.amazonaws.com,g" /etc/apt/sources.list
3.) Set your apt debug levels to whatever you want
4) Run "apt-get -y update"
5) Run "apt-get -y install firefox xorg libreoffice > /tmp/install.log 2>&1"

The S3 EC2 mirrors seem to reproduce it more reliably for some reason. I thought that this was an issue with S3, but in attempting to prove to Amazon that is is S3, I found the evidence indicating that the cached files are being misnamed.

Confirmed on Natty:

Fetched 13.7 MB in 12s (1,106 kB/s)
W: Failed to fetch bzip2:/var/lib/apt/lists/partial/us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com_ubuntu_dists_natty-updates_main_binary-i386_Packages Hash Sum mismatch

W: Failed to fetch bzip2:/var/lib/apt/lists/partial/us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com_ubuntu_dists_natty-updates_universe_binary-i386_Packages Hash Sum mismatch

W: Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/dists/natty-updates/main/i18n/Index No Hash entry in Release file /var/lib/apt/lists/partial/us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com_ubuntu_dists_natty-updates_main_i18n_Index

root@ip-10-117-77-162:/var/lib/apt/lists/partial# md5sum *
66b1e003087af95e4c3ecb2595888723 us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com_ubuntu_dists_natty-updates_main_binary-i386_Packages
 -- Matches universe/source/Sources.bz2
6c0170a630ab2c2526fce1a15a557cc7 us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com_ubuntu_dists_natty-updates_main_binary-i386_Packages.decomp.FAILED
 -- Matches universe/sources/Sources
27bb09fcfc7a4dee731a84d32ab59561 us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com_ubuntu_dists_natty-updates_main_i18n_Index
 -- Matches universe/binary-i386/Packages.bz2
5238567412ee16c16aa4dfb1282e4baa us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com_ubuntu_dists_natty-updates_universe_binary-i386_Packages
 -- Matches main/binary-i386/Packages.bz2
57a2cf0d9cf10366d3905285f3507227 us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com_ubuntu_dists_natty-updates_universe_binary-i386_Packages.decomp.FAILED
 -- Matches main/binary-i386/Packages

From natty-updates Apt Sources:
 5238567412ee16c16aa4dfb1282e4baa 431291 main/binary-i386/Packages.bz2
 c757bc649fcb9fd50d9c5b895727daa8 554975 main/binary-i386/Packages.gz
 d7f12a4b835c3837b80f214b1dd6753e 102 main/binary-i386/Release
 57a2cf0d9cf10366d3905285f3507227 3333025 main/binary-i386/Packages
 f12dc36fa5dd364ae0b7776ba4bafa28 183259 universe/binary-i386/Packages.gz
 9f930c4d57ab8f70aab2d36f06a27b3d 779085 universe/binary-i386/Packages
 0951480c2c33548ec3618ab1082b77d9 106 universe/binary-i386/Release
 27bb09fcfc7a4dee731a84d32ab59561 138336 universe/binary-i386/Packages.bz2
 6c0170a630ab2c2526fce1a15a557cc7 149525 universe/source/Sources
 46cd339686c8e38c23a541190b436a29 46951 universe/source/Sources.gz
 66b1e003087af95e4c3ecb2595888723 40485 universe/source/Sources.bz2
 a4245d47dc44f65f4ce1786652c27c3d 108 universe/source/Release

And Curl headers:
root@ip-10-117-77-162:/var/lib/apt/lists/partial# curl -I http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/dists/natty-updates/main/binary-i386/Packages.bz2
HTTP/1.1 200 OK
x-amz-id-2: HmP/KlL95l4732MGdCO2Yuz/Sx40eMJ0Z3XYspztQra8J9PjkNxajSKT/l4ZcQan
x-amz-request-id: CA9A66000053719F
Date: Tue, 06 Mar 2012 22:42:08 GMT
Last-Modified: Tue, 06 Mar 2012 20:56:01 GMT
ETa...

Status changed to 'Confirmed' because the bug affects multiple users.

Bug confirmed for Oneiric too:

root@ip-10-64-61-11:/var/cache/apt/archives/partial# dpkg -I xserver-xorg-core_2%3a1.10.4-1ubuntu4.2_i386.deb
 new debian package, version 2.0.
 size 90230 bytes: control archive= 720 bytes.
     709 bytes, 18 lines control
     255 bytes, 3 lines md5sums
 Package: xserver-xorg-video-mach64
 Version: 6.9.0-1
 Architecture: i386
 Maintainer: Ubuntu Developers <email address hidden>
 Original-Maintainer: Debian X Strike Force <email address hidden>
 Installed-Size: 268
 Depends: libc6 (>= 2.4), xorg-video-abi-10, xserver-xorg-core (>= 2:1.10.0-0ubuntu1~)
 Provides: xorg-driver-video
 Section: x11
 Priority: optional
 Description: X.Org X server -- ATI Mach64 display driver
  This driver for the X.Org X server (see xserver-xorg for a further description)
  provides support for the ATI Mach64 series.
  .
  More information about X.Org can be found at:
  <URL:http://www.X.org>
  .
  This package is built from the X.org xf86-video-mach64 driver module.
root@ip-10-64-61-11:/var/cache/apt/archives/partial#

root@ip-10-64-61-11:/var/cache/apt/archives/partial# dpkg -I xserver-xorg-video-mach64_6.9.0-1_i386.deb
 new debian package, version 2.0.
 size 6706 bytes: control archive= 977 bytes.
    1206 bytes, 25 lines control
     217 bytes, 3 lines md5sums
 Package: xserver-xorg-video-ati
 Version: 1:6.14.99~git20110811.g93fc084-0ubuntu1
 Architecture: i386
 Maintainer: Ubuntu Developers <email address hidden>
 Installed-Size: 96
 Depends: libc6 (>= 2.1.3), libpciaccess0, xorg-video-abi-10, xserver-xorg-core (>= 2:1.10.0-0ubuntu1~), xserver-xorg-video-r128, xserver-xorg-video-mach64, xserver-xorg-video-radeon
 Provides: xorg-driver-video
 Section: x11
 Priority: optional
 Description: X.Org X server -- AMD/ATI display driver wrapper
  This package provides the 'ati' driver for the AMD/ATI Mach64, Rage128,
  Radeon, FireGL, FireMV, FirePro and FireStream series. This driver is
  actually a wrapper that loads one of the 'mach64', 'r128' or 'radeon'
  sub-drivers depending on the hardware.
  These sub-drivers are brought through package dependencies.
  .
  Users of Rage, Mach, or Radeon boards may remove this package only if
  they use Driver "r128", "mach64", or "radeon" in /etc/X11/xorg.conf
  instead of relying on autodetection.
  .
  More information about X.Org can be found at:
  <URL:http://www.X.org>
  .
  This package is built from the X.org xf86-video-ati driver module.
 Original-Maintainer: Debian X Strike Force <email address hidden>

root@ip-10-64-61-11:/var/cache/apt/archives/partial# dpkg -I xserver-xorg-video-ati_1%3a6.14.99~git20110811.g93fc084-0ubuntu1_i386.deb
 new debian package, version 2.0.
 size 12712 bytes: control archive= 785 bytes.
     793 bytes, 20 lines control
     316 bytes, 4 lines md5sums
 Package: xserver-xorg-video-fbdev
 Version: 1:0.4.2-3ubuntu6
 Architecture: i386
 Maintainer: Ubuntu Developers <email address hidden>
 Installed-Size: 96
 Depends: libc6 (>= 2.1.3), xorg-video-abi-10, xserver-xorg...

I've confirmed this bug exists on Hardy, Lucid, Maverick, Oneiric, Natty and Precise.

Ben,
  Can you see this fail on mirrors other than the new s3 backed mirrors?
  I've tried several times and have been unable to reproduce on the us-east-1.ec2.archive.ubuntu.com mirrors.
  However, I can reproduce fairly regularly against s3 mirrors

  See log at http://paste.ubuntu.com/872478/
  The command I was using was:

 ( sudo apt-get clean; sudo apt-get update --assume-yes ; sudo apt-get --download-only -y install -o=Debug::Acquire::http=true --assume-yes firefox ; echo $? ) </dev/null 2>&1 | tee /tmp/apt-download.log

  r=0; log=/tmp/apt-download.log; pkg="ubuntu-desktop";
  while [ "$r" = "0" ]; do
    ( sudo apt-get clean; sudo apt-get update --assume-yes &&
      sudo apt-get --download-only -y install -o=Debug::Acquire::http=true --assume-yes $pkg ;
      echo $? ) </dev/null 2>&1 | tee $log
    tail -n 1 $log > out; read r < out ;
    echo "$(date): finished, $r";
   done

Then, I've got this running in a loop from a precise m1.large in us-east-1:

r=0; log=/tmp/apt-download.log;
while [ $r -eq 0 ]; do
  ( sudo apt-get clean; sudo apt-get update --assume-yes && sudo apt-get --downl
oad-only -y install -o=Debug::Acquire::http=true --assume-yes ubuntu-desktop ; echo $? ) </dev/null 2>&1 | tee $log ;
  tail -n 1 $log > out; read r < out ;

  Until we can see it fail against more traditional http server, we almost have to suspect failure of S3 mirrors. Anything
 "$(date): finished, $r"; done

Some more data, after discussing with slangasek.
The attached script will run seemingly forever (100 times run) as it is, however if we remove the flag:
  -o=Acquire::http::Pipeline-Depth=0
to apt, then it will generally fail < 10 times.
(data collected on m1.large "right now", with mirrors set to http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/).

It would seem to me that the Pipeline-Depth is the source of the issue. See [1] for apt.conf description, and [2] for a quick google of 's3' 'http' 'pipeline' that gives some evidence that Pipeline and S3 aren't friends.

--
[1] http://manpages.ubuntu.com/manpages/precise/en/man5/apt.conf.5.html
[2] http://stackoverflow.com/questions/7752802/does-s3-support-http-pipeling

Scott,

I can confirm that "-o=Acquire::http::Pipeline-Depth=0" appears to fix the problem. We'll need to look at putting that configuration option into the cloud-images.

Thanks Scott and Steve for digging on this.

On Wed, 7 Mar 2012, Ben Howard wrote:

> Scott,
>
> I can confirm that "-o=Acquire::http::Pipeline-Depth=0" appears to fix
> the problem. We'll need to look at putting that configuration option
> into the cloud-images.

Well.... thats not really a complete fix though.
 * existing images do not have that setting in them, and they will have to
   'apt-get update && apt-get upgrade' to *get* that setting (whch can
   fail).
 * other images out there would not have this setting, and would just
   start to fail when we switch DNS records to S3 mirrors.
 * other people building images would need to know they need to do this.
   (this could be alleviated by it being a file laid down by a common
   package like cloud-init, but even then, we'd have to SRU the fix
   to old releases.)

I'm really not sure what to do. We dont want EC2 laden with "gotchas"
that only work if you use our special images. While I certainly want our
images to work, I don't want to just make like for difficult for others
who are not using them.

Regarding comment #13, I think the concern is moot right now. I was able to replicate the issue with "-o=Acquire::http::Pipeline-Depth=0" turned on. Which means that pipelining may contribute but doesn't appear to be the sole cause.

Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/libx/libxinerama/libxinerama1_1.1.1-3build1_amd64.deb Size mismatch
Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/libx/libxrandr/libxrandr2_1.3.2-2_amd64.deb Size mismatch
Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/s/shared-mime-info/shared-mime-info_1.0-0ubuntu1_amd64.deb Size mismatch
Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/g/gtk+2.0/libgtk2.0-0_2.24.10-0ubuntu5_amd64.deb Size mismatch
Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/x/xorg/x11-common_7.6+10ubuntu1_all.deb Size mismatch
Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/libi/libice/libice6_1.0.7-2build1_amd64.deb Size mismatch
Failed to fetch http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/pool/main/p/policykit-1/libpolkit-agent-1-0_0.104-1_amd64.deb Size mismatch
Fetched 28.1 MB in 8s (3,261 kB/s)
E: Some files failed to download

root@ip-10-80-113-210:/var/cache/apt/archives/partial# dpkg -I libxinerama1_2%3a1.1.1-3build1_amd64.deb
 new debian package, version 2.0.
 size 456104 bytes: control archive= 1639 bytes.
     805 bytes, 18 lines control
    1330 bytes, 16 lines md5sums
     479 bytes, 20 lines * postinst #!/bin/sh
     376 bytes, 19 lines * postrm #!/bin/sh
     194 bytes, 14 lines * prerm #!/bin/sh
      34 bytes, 1 lines triggers
 Package: shared-mime-info
 Version: 1.0-0ubuntu1
 Architecture: amd64
 Maintainer: Ubuntu Developers <email address hidden>
 Installed-Size: 2188
 Depends: libc6 (>= 2.3), libglib2.0-0 (>= 2.24.0), libxml2 (>= 2.7.4)
 Conflicts: libglib2.0-0 (<< 2.17.2), libgnomevfs2-0 (<< 1:2.24.0), tracker (<< 0.6.90)
 Section: misc
 Priority: optional
 Multi-Arch: foreign
 Homepage: http://freedesktop.org/wiki/Software/shared-mime-info
 Description: FreeDesktop.org shared MIME database and spec
  This is the shared MIME-info database from the X Desktop Group. It is required
  by any program complying to the Shared MIME-Info Database spec, which is also
  included in this package.
  .
  At this time at least ROX, GNOME, KDE and XFCE use this database.
 Original-Maintainer: Sebastian Dröge <email address hidden>

           One setting is provided to control the pipeline depth in cases
           where the remote server is not RFC conforming or buggy (such as
           Squid 2.0.2). Acquire::http::Pipeline-Depth can be a value from 0
           to 5 indicating how many outstanding requests APT should send. A
           value of zero MUST be specified if the remote host does not
           properly linger on TCP connections - otherwise data corruption will
           occur. Hosts which require this are in violation of RFC 2068.

I guess an alternative to trying to SRU this everywhere it affects images would be to ask Amazon to support RFC 2068?

Scratch my previous comment in #14, it was a test error.

I'm also seeing this too on preceise.

I clicked 'Post Comment' too fast. I'm seeing this when I do an apt-get update.

W: Failed to fetch bzip2:/var/lib/apt/lists/partial/mirror.csclub.uwaterloo.ca_ubuntu_dists_precise_main_source_Sources Hash Sum mismatch

W: Failed to fetch bzip2:/var/lib/apt/lists/partial/mirror.csclub.uwaterloo.ca_ubuntu_dists_precise_universe_binary-amd64_Packages Hash Sum mismatch

W: Failed to fetch bzip2:/var/lib/apt/lists/partial/mirror.csclub.uwaterloo.ca_ubuntu_dists_precise_main_binary-i386_Packages Hash Sum mismatch

W: Failed to fetch bzip2:/var/lib/apt/lists/partial/mirror.csclub.uwaterloo.ca_ubuntu_dists_precise_universe_binary-i386_Packages Hash Sum mismatch

Steve,

We've been able to prove that disabling HTTP pipelining fixes the problem, but the question remains:

How come package-A.deb has package-B's contents? The downloaded files are whole and match the proper checksums, but they have the wrong content. If I understand comment #15 right, shouldn't the content fail to checksum properly?

Apparently, my problems have cleared up. Not sure why.

The problem here is that S3 is broken regarding HTTP pipeline responses. What is happening is that the client is that apt is requesting files A, B, C, D in the same requests and S3 appears to be responding with files B, C, D, A (or some simularly arbitrary order). The result is that apt saves the data under the wrong file name.

Here is a summary of a packet trace (edited for readability). S3 uses the "ETag" header to store the MD5 of the file. So using the the Etag, you can conclude what file contents are.

GET /ubuntu/pool/main/libx/libxdamage/libxdamage1_1.1.3-2build1_amd64.deb HTTP/1.1
GET /ubuntu/pool/main/libx/libxfixes/libxfixes3_5.0-4ubuntu1_amd64.deb HTTP/1.1
GET /ubuntu/pool/main/libx/libxxf86vm/libxxf86vm1_1.1.1-2build1_amd64.deb HTTP/1.1
GET /ubuntu/pool/main/m/mesa/libgl1-mesa-glx_8.0.1-0ubuntu2_amd64.deb HTTP/1.1
GET /ubuntu/pool/main/g/gtk+2.0/libgtk2.0-common_2.24.10-0ubuntu5_all.deb HTTP/1.1
GET /ubuntu/pool/main/libx/libxcomposite/libxcomposite1_0.4.3-2build1_amd64.deb HTTP/1.1
GET /ubuntu/pool/main/libx/libxcursor/libxcursor1_1.1.12-1_amd64.deb HTT
GET /ubuntu/pool/main/libx/libxi/libxi6_1.5.99.3-0ubuntu2_amd64.deb HTTP/1.1
GET /ubuntu/pool/main/libx/libxinerama/libxinerama1_1.1.1-3build1_amd64.deb HTTP/1.1
        6e773d7e05527fa35a201209660b517c - libxfixes3
            x-amz-id-2: EgVy5vizxmUOYRxfyjFXJzt7k30EVGtn2gyZ1q0P/jN3ROZS+3KjUEwSzX3JFJKE
            x-amz-request-id: 8CD6DF060D72CFDA
        c06221dc919029cf8482198e724fa111 - libxxf86vm1
            x-amz-id-2: xr56YqhmQzov1YP+Oev9lnpKQtOYX98DkrJD7aWkSwQq8Ke77+4PD4yHiRvcBo6w
            x-amz-request-id: 67B263265E35A521
GET /ubuntu/pool/main/g/gtk+2.0/libgtk2.0-0_2.24.10-0ubuntu5_amd64.deb HTTP/1.1
        a47eb17218d742a4ceae59cc08e80d8c - libgl1-mesa-glx
            x-amz-id-2: Ou92TrRA7dfHyF97UEfO0TvQ9XI9vOxYj0LgWCR3GXFHMhAZ7nzQ71cGUqMQHI1X
            x-amz-request-id: C9393E43C43C6471
        90787d373defad09a6dd99db3c3e7614 - libgtk2.0-common
            x-amz-id-2: ze4hg1y4hT8+ZXXzY7JHRclypybdYj+HIQEG6r2eulAt+twaZn/GG+y6ifi9MnL8
            x-amz-request-id: 5DDC52A0301600FC
        77440ea3c635a3243cc3f6bc98305f22 - libxcomposite1
            x-amz-id-2: +hzkmvonwW+1nh4gcsQ/qXsmXUq2CUj4hAQ69EjqAEb6MVOpxHvPnh8l7Qh2Wc5Q
            x-amz-request-id: 54FDE3289C1021F3
        f5754520b029296a6f0a5a14a1dd4b5a - libxcursor1
            x-amz-id-2: WvhZ9o1R3kMvsFEmERvI2V/eOY+0ksHnvs1suVCX1Es8aa/XxyU7drgvj0Bj9zHj
            x-amz-request-id: 9F408ACCBFFA66E5
        728ef8819d6f3ec4b827d887951a86ce - libxi6
            x-amz-id-2: Tgqse2GgWGu+OEVzv/atXhjP6FdszvjVeFGic3gq5LxGUXDFVS5YdF+CGFtF7dkC
            x-amz-request-id: C57783745F1192CE
        90b84d263ad2b7f76868ce02967c2107 - libxinerama1
            x-amz-id-2: 1Mz7B1r13S1NnNIingOxj2iOlQq8a5arP7K/Wt/zgd+NL78Q73/MRDzFhPf4kXFL
            x-amz-request-id: DB666BE3004C6CD9
        652e1f2550726fc72fa432413e6e287b - libxdamage1
            x-amz-id-2: X5TQqCJey0xtne7jKZYlX+8c5AqRM2aTVD1cvEHWPIx4qcc8L9rvyqlG0FJcPT4g
            x-amz-request-id: 74A2EB9093B4F00A
        c8f474321079048af432d47d66a4afb4 - libgtk2.0-0
            x-amz-id-2: Fgkgv/JY7WnSojk+pBeQ430gNCvjRvXG1KZzrbQfEG2j/jqPvvcFVRS07ZBMsFa9
  ...

Note that this issue with HTTP pipelining is known to apt upstream, and documented in the apt.conf(5) manpage - this is exactly why the commandline option exists to toggle the behavior, basically.

           One setting is provided to control the pipeline depth in cases
           where the remote server is not RFC conforming or buggy (such as
           Squid 2.0.2). Acquire::http::Pipeline-Depth can be a value from 0
           to 5 indicating how many outstanding requests APT should send. A
           value of zero MUST be specified if the remote host does not
           properly linger on TCP connections - otherwise data corruption will
           occur. Hosts which require this are in violation of RFC 2068.

Ben and I did backports, and I've uploaded to lucid, maverick, natty, oneiric.
We're waiting on approval for opulation into -proposed.

The plan from there is to get testing and population into -updates.
Then,
  * release new images including this modification
  * give some time for existing instances to populate
  * email to different lists announcing need for the update/change
  * switch S3 mirrors into production.

These are all uploaded and waiting:
 * lucid: https://launchpad.net/ubuntu/lucid/+queue?queue_state=1&queue_text=cloud-init
 * maverick: https://launchpad.net/ubuntu/maverick/+queue?queue_state=1&queue_text=cloud-init
 * natty: https://launchpad.net/ubuntu/natty/+queue?queue_state=1&queue_text=cloud-init
 * oneiric: https://launchpad.net/ubuntu/oneiric/+queue?queue_state=1&queue_text=cloud-init

I notice this bug is reported against both apt and cloud-init, but the patches are only for cloud-init. Are there any changes required of apt? If not, perhaps those tasks can all be closed out?

Hello Ben, or anyone else affected,

Accepted cloud-init into oneiric-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Hello Ben, or anyone else affected,

Accepted cloud-init into natty-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Hello Ben, or anyone else affected,

Accepted cloud-init into maverick-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

FYI, I accepted the maverick one, even though it is EOL very soon, thinking that it may be useful for generating one last AMI which people will use to migrate to natty. If no new cloud image will be produced, I don't see any reason to actually move that to maverick-updates.

Re #29: I plan on spinning a final Maverick on its EOF for historical reasons. The other reason why I would like to see Maverick get this to updates is for people who upgrade from Maverick to something newer.

Re #27: I've confirmed the fixes in proposed work. We're good to go here.

This bug was fixed in the package cloud-init - 0.5.10-0ubuntu1.7

---------------
cloud-init (0.5.10-0ubuntu1.7) lucid-proposed; urgency=low

  * add ability to configure Acquire::http::Pipeline-Depth via
    cloud-config setting 'apt_pipelining' (LP: #948461)
  * debian/cloud-init.postinst: address population of apt_pipeline
    setting on installation.
 -- Scott Moser <email address hidden> Fri, 16 Mar 2012 14:32:50 -0400

This bug was fixed in the package cloud-init - 0.6.1-0ubuntu8.1

---------------
cloud-init (0.6.1-0ubuntu8.1) natty-proposed; urgency=low

  * add ability to configure Acquire::http::Pipeline-Depth via
    cloud-config setting 'apt_pipelining' (LP: #948461)
  * debian/cloud-init.postinst: address population of apt_pipeline
    setting on installation.
 -- Ben Howard <email address hidden> Fri, 16 Mar 2012 11:05:48 -0600

This bug was fixed in the package cloud-init - 0.6.1-0ubuntu22.1

---------------
cloud-init (0.6.1-0ubuntu22.1) oneiric-proposed; urgency=low

  * add ability to configure Acquire::http::Pipeline-Depth via
    cloud-config setting 'apt_pipelining' (LP: #948461)
  * debian/cloud-init.postinst: address population of apt_pipeline
    setting on installation.
 -- Ben Howard <email address hidden> Fri, 16 Mar 2012 15:44:50 -0600

This bug was fixed in the package cloud-init - 0.5.15-0ubuntu3.1

---------------
cloud-init (0.5.15-0ubuntu3.1) maverick-proposed; urgency=low

  * add ability to configure Acquire::http::Pipeline-Depth via
    cloud-config setting 'apt_pipelining' (LP: #948461)
  * debian/cloud-init.postinst: address population of apt_pipeline
    setting on installation.
 -- Scott Moser <email address hidden> Fri, 16 Mar 2012 14:36:07 -0400

this has been mitigated by applying a fix to the EC2 images.

apt (Ubuntu Precise)

running Ubuntu in VM-ware through a proxy and a Secure-Web-Gateway.

To disable the http pipeline when doing a dist-upgrade did the trick for me.
(temporary) sudo apt-get -o Acquire::http::Pipeline-Depth="0" -y dist-upgrade
(permanent) echo 'Acquire::http::Pipeline-Depth "0";' | sudo tee /etc/apt/apt.conf.d/99-no-pipelining

Hardy has seen the end of its life and is no longer receiving any updates. Marking the Hardy task for this ticket as "Won't Fix".

This was fixed in 1.1 or later, not entirely sure about the exact version.