Comment 21 for bug 948461

The problem here is that S3 is broken regarding HTTP pipeline responses. What is happening is that the client is that apt is requesting files A, B, C, D in the same requests and S3 appears to be responding with files B, C, D, A (or some simularly arbitrary order). The result is that apt saves the data under the wrong file name.

Here is a summary of a packet trace (edited for readability). S3 uses the "ETag" header to store the MD5 of the file. So using the the Etag, you can conclude what file contents are.

GET /ubuntu/pool/main/libx/libxdamage/libxdamage1_1.1.3-2build1_amd64.deb HTTP/1.1
GET /ubuntu/pool/main/libx/libxfixes/libxfixes3_5.0-4ubuntu1_amd64.deb HTTP/1.1
GET /ubuntu/pool/main/libx/libxxf86vm/libxxf86vm1_1.1.1-2build1_amd64.deb HTTP/1.1
GET /ubuntu/pool/main/m/mesa/libgl1-mesa-glx_8.0.1-0ubuntu2_amd64.deb HTTP/1.1
GET /ubuntu/pool/main/g/gtk+2.0/libgtk2.0-common_2.24.10-0ubuntu5_all.deb HTTP/1.1
GET /ubuntu/pool/main/libx/libxcomposite/libxcomposite1_0.4.3-2build1_amd64.deb HTTP/1.1
GET /ubuntu/pool/main/libx/libxcursor/libxcursor1_1.1.12-1_amd64.deb HTT
GET /ubuntu/pool/main/libx/libxi/libxi6_1.5.99.3-0ubuntu2_amd64.deb HTTP/1.1
GET /ubuntu/pool/main/libx/libxinerama/libxinerama1_1.1.1-3build1_amd64.deb HTTP/1.1
        6e773d7e05527fa35a201209660b517c - libxfixes3
            x-amz-id-2: EgVy5vizxmUOYRxfyjFXJzt7k30EVGtn2gyZ1q0P/jN3ROZS+3KjUEwSzX3JFJKE
            x-amz-request-id: 8CD6DF060D72CFDA
        c06221dc919029cf8482198e724fa111 - libxxf86vm1
            x-amz-id-2: xr56YqhmQzov1YP+Oev9lnpKQtOYX98DkrJD7aWkSwQq8Ke77+4PD4yHiRvcBo6w
            x-amz-request-id: 67B263265E35A521
GET /ubuntu/pool/main/g/gtk+2.0/libgtk2.0-0_2.24.10-0ubuntu5_amd64.deb HTTP/1.1
        a47eb17218d742a4ceae59cc08e80d8c - libgl1-mesa-glx
            x-amz-id-2: Ou92TrRA7dfHyF97UEfO0TvQ9XI9vOxYj0LgWCR3GXFHMhAZ7nzQ71cGUqMQHI1X
            x-amz-request-id: C9393E43C43C6471
        90787d373defad09a6dd99db3c3e7614 - libgtk2.0-common
            x-amz-id-2: ze4hg1y4hT8+ZXXzY7JHRclypybdYj+HIQEG6r2eulAt+twaZn/GG+y6ifi9MnL8
            x-amz-request-id: 5DDC52A0301600FC
        77440ea3c635a3243cc3f6bc98305f22 - libxcomposite1
            x-amz-id-2: +hzkmvonwW+1nh4gcsQ/qXsmXUq2CUj4hAQ69EjqAEb6MVOpxHvPnh8l7Qh2Wc5Q
            x-amz-request-id: 54FDE3289C1021F3
        f5754520b029296a6f0a5a14a1dd4b5a - libxcursor1
            x-amz-id-2: WvhZ9o1R3kMvsFEmERvI2V/eOY+0ksHnvs1suVCX1Es8aa/XxyU7drgvj0Bj9zHj
            x-amz-request-id: 9F408ACCBFFA66E5
        728ef8819d6f3ec4b827d887951a86ce - libxi6
            x-amz-id-2: Tgqse2GgWGu+OEVzv/atXhjP6FdszvjVeFGic3gq5LxGUXDFVS5YdF+CGFtF7dkC
            x-amz-request-id: C57783745F1192CE
        90b84d263ad2b7f76868ce02967c2107 - libxinerama1
            x-amz-id-2: 1Mz7B1r13S1NnNIingOxj2iOlQq8a5arP7K/Wt/zgd+NL78Q73/MRDzFhPf4kXFL
            x-amz-request-id: DB666BE3004C6CD9
        652e1f2550726fc72fa432413e6e287b - libxdamage1
            x-amz-id-2: X5TQqCJey0xtne7jKZYlX+8c5AqRM2aTVD1cvEHWPIx4qcc8L9rvyqlG0FJcPT4g
            x-amz-request-id: 74A2EB9093B4F00A
        c8f474321079048af432d47d66a4afb4 - libgtk2.0-0
            x-amz-id-2: Fgkgv/JY7WnSojk+pBeQ430gNCvjRvXG1KZzrbQfEG2j/jqPvvcFVRS07ZBMsFa9
            x-amz-request-id: 57FF62A675D21685
GET /ubuntu/pool/main/g/gtk+2.0/libgtk2.0-bin_2.24.10-0ubuntu5_amd64.deb HTTP/1.1
        e03a09b33f955e8da797a2f33fbf3423 - libgtk2.0-bin
            x-amz-id-2: RaJe83jyFhTwQdvnp6ZjYJQMwKOIengoH2U//VnJZxgagvTPmGYMsAwKGkjGh+Mz
            x-amz-request-id: B9D4754504E9CA3E