Linaro Android Infrastructure

Bug #932088
Comment #56

Comment 56 for bug 932088

Revision history for this message

Deepti B. Kalakeri (deeptik) wrote on 2012-06-06: Re: [Bug 932088] Re: Ubuntu EC2 package mirror intermitent failures

#56

On Wed, Jun 6, 2012 at 12:17 PM, Paul Sokolovsky
<email address hidden>wrote:

> Today we're having 403's on ci.linaro.org again, see attachment for a
> sample log.
>
>

** Attachment added: "i-18e33761.log"
>
> https://bugs.launchpad.net/linaro-android-infrastructure/+bug/932088/+attachment/3177116/+files/i-18e33761.log
>

Right, I am looking into it.
Between, you mention in the comment #53 you have disabled pipelining, but
a-b* slave log shows something like

+ + echo Acquire::http::Pipeline-Depth 0;^M
tee /etc/apt/apt.conf.d/99no-pipelining^M
Acquire::http::Pipeline-Depth 0;^M

Is this something different than what you mentioned about pipelining ?
Also, am trying to access the s3 mirror as below instead of just
us-east-1.ec2.archive.ubuntu.com and I still getting 403 error.

$wget
http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/dists/precise-updates/universe/binary-i386/Packages
--2012-06-06 07:11:27--
http://us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com/ubuntu/dists/precise-updates/universe/binary-i386/Packages
Resolving us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com... 72.21.211.168
Connecting to us-east-1.ec2.archive.ubuntu.com.s3.amazonaws.com|72.21.211.168|:80...
connected.
HTTP request sent, awaiting response... 403 Forbidden
2012-06-06 07:11:27 ERROR 403: Forbidden.

so, what are the exact changes on a-b* that we have that fixes this problem
?

>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/932088
>
> Title:
> Ubuntu EC2 package mirror intermitent failures
>
> Status in Linaro Android Infrastructure:
> Fix Released
> Status in Linaro Continuous Integration:
> New
> Status in Ubuntu:
> Confirmed
>
> Bug description:
> Not too frequent, but regularly (say, once a month) https://android-
> build.linaro.org/ service, which is Jenkins with EC2 slaves, gets
> failures when initializing EC2 instances, which involves installing
> packages.
>
> The log looks like:
> Connecting to ec2-23-20-20-233.compute-1.amazonaws.com on port 22.
> Waiting for SSH to come up. Sleeping 5.
> Connecting to ec2-23-20-20-233.compute-1.amazonaws.com on port 22.
> Connected via SSH.
> Authenticating as ubuntu
> Connecting to ec2-23-20-20-233.compute-1.amazonaws.com on port 22.
> Connected via SSH.
> Executing init script
> + apt-get update
> Ign http://security.ubuntu.com natty-security InRelease
> Get:1 http://security.ubuntu.com natty-security Release.gpg [198 B]
> Get:2 http://security.ubuntu.com natty-security Release [39.8 kB]
> Ign http://us-east-1.ec2.archive.ubuntu.com natty InRelease
> Ign http://us-east-1.ec2.archive.ubuntu.com natty-updates InRelease
> Ign http://us-east-1.ec2.archive.ubuntu.com natty Release.gpg
> Ign http://us-east-1.ec2.archive.ubuntu.com natty-updates Release.gpg
> Ign http://us-east-1.ec2.archive.ubuntu.com natty Release
> Ign http://us-east-1.ec2.archive.ubuntu.com natty-updates Release
> Ign http://us-east-1.ec2.archive.ubuntu.com natty/main amd64
> Packages/DiffIndex
> Ign http://us-east-1.ec2.archive.ubuntu.com natty/universe amd64
> Packages/DiffIndex
> Ign http://us-east-1.ec2.archive.ubuntu.com natty/main TranslationIndex
> Ign http://us-east-1.ec2.archive.ubuntu.com natty/universe
> TranslationIndex
> Ign http://us-east-1.ec2.archive.ubuntu.com natty-updates/main amd64
> Packages/DiffIndex
> Ign http://us-east-1.ec2.archive.ubuntu.com natty-updates/universe amd64
> Packages/DiffIndex
> Ign http://us-east-1.ec2.archive.ubuntu.com natty-updates/main
> TranslationIndex
> Ign http://us-east-1.ec2.archive.ubuntu.com natty-updates/universe
> TranslationIndex
> Err http://us-east-1.ec2.archive.ubuntu.com natty/main Sources
> 403 Forbidden [IP: 10.210.205.172 80]
> Err http://us-east-1.ec2.archive.ubuntu.com natty/universe Sources
> 403 Forbidden [IP: 10.210.205.172 80]
> ...
>
>
> Trying to access a specific file manually shows that there're
> interleaving 404 and 403 responses for it (the file indeed doesn't exists).
> And generally situation look like 403 is being intermittently returned for
> any url (be it 200 or 404 one).
>
> ubuntu@ip-10-243-34-224:~$ wget
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
> --2012-02-14 13:42:12--
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
> Resolving us-east-1.ec2.archive.ubuntu.com... 10.250.142.223,
> 10.252.111.96, 10.202.26.15, ...
> Connecting to us-east-1.ec2.archive.ubuntu.com|10.250.142.223|:80...
> connected.
> HTTP request sent, awaiting response... 404 Not Found
> 2012-02-14 13:42:12 ERROR 404: Not Found.
>
> ubuntu@ip-10-243-34-224:~$ wget
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
> --2012-02-14 13:42:14--
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
> Resolving us-east-1.ec2.archive.ubuntu.com... 10.210.205.172,
> 10.250.142.223, 10.252.111.96, ...
> Connecting to us-east-1.ec2.archive.ubuntu.com|10.210.205.172|:80...
> connected.
> HTTP request sent, awaiting response... 403 Forbidden
> 2012-02-14 13:42:14 ERROR 403: Forbidden.
>
> ubuntu@ip-10-243-34-224:~$ wget
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
> --2012-02-14 13:42:16--
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
> Resolving us-east-1.ec2.archive.ubuntu.com... 10.202.26.15,
> 10.210.205.172, 10.250.142.223, ...
> Connecting to us-east-1.ec2.archive.ubuntu.com|10.202.26.15|:80...
> connected.
> HTTP request sent, awaiting response... 404 Not Found
> 2012-02-14 13:42:16 ERROR 404: Not Found.
>
> ubuntu@ip-10-243-34-224:~$ wget
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
> --2012-02-14 13:42:17--
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
> Resolving us-east-1.ec2.archive.ubuntu.com... 10.252.111.96,
> 10.202.26.15, 10.210.205.172, ...
> Connecting to us-east-1.ec2.archive.ubuntu.com|10.252.111.96|:80...
> connected.
> HTTP request sent, awaiting response... 404 Not Found
> 2012-02-14 13:42:17 ERROR 404: Not Found.
>
> ubuntu@ip-10-243-34-224:~$ wget
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
> --2012-02-14 13:42:18--
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
> Resolving us-east-1.ec2.archive.ubuntu.com... 10.250.142.223,
> 10.252.111.96, 10.202.26.15, ...
> Connecting to us-east-1.ec2.archive.ubuntu.com|10.250.142.223|:80...
> connected.
> HTTP request sent, awaiting response... 404 Not Found
> 2012-02-14 13:42:18 ERROR 404: Not Found.
>
> ubuntu@ip-10-243-34-224:~$ wget
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
> --2012-02-14 13:42:19--
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
> Resolving us-east-1.ec2.archive.ubuntu.com... 10.210.205.172,
> 10.250.142.223, 10.252.111.96, ...
> Connecting to us-east-1.ec2.archive.ubuntu.com|10.210.205.172|:80...
> connected.
> HTTP request sent, awaiting response... 403 Forbidden
> 2012-02-14 13:42:19 ERROR 403: Forbidden.
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/linaro-android-infrastructure/+bug/932088/+subscriptions
>

--
Thanks and Regards,
Deepti

On Wed, Jun 6, 2012 at 12:17 PM, Paul Sokolovsky
<932088@bugs.launchpad.net>wrote:

> Today we're having 403's on ci.linaro.org again, see attachment for a
> sample log.
>
>

** Attachment added: "i-18e33761.log"
>
> https://bugs.launchpad.net/linaro-android-infrastructure/+bug/932088/+attachment/3177116/+files/i-18e33761.log
>

Right, I am looking into it.
Between, you mention in the comment #53 you have disabled pipelining, but
a-b* slave log shows something like

+ + echo Acquire::http::Pipeline-Depth 0;^M
tee /etc/apt/apt.conf.d/99no-pipelining^M
Acquire::http::Pipeline-Depth 0;^M

Is this something different than what you mentioned about pipelining ?
Also, am trying to access the s3 mirror as below instead of  just
us-east-1.ec2.archive.ubuntu.com and I still getting 403 error.

so, what are the exact changes on a-b* that we have that fixes this problem
?

>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/932088
>
> Title:
>  Ubuntu EC2 package mirror intermitent failures
>
> Status in Linaro Android Infrastructure:
>  Fix Released
> Status in Linaro Continuous Integration:
>  New
> Status in Ubuntu:
>  Confirmed
>
> Bug description:
>  Not too frequent, but regularly (say, once a month) https://android-
>  build.linaro.org/ service, which is Jenkins with EC2 slaves, gets
>  failures when initializing EC2 instances, which involves installing
>  packages.
>
>  The log looks like:
>  Connecting to ec2-23-20-20-233.compute-1.amazonaws.com on port 22.
>  Waiting for SSH to come up. Sleeping 5.
>  Connecting to ec2-23-20-20-233.compute-1.amazonaws.com on port 22.
>  Connected via SSH.
>  Authenticating as ubuntu
>  Connecting to ec2-23-20-20-233.compute-1.amazonaws.com on port 22.
>  Connected via SSH.
>  Executing init script
>  + apt-get update
>  Ign http://security.ubuntu.com natty-security InRelease
>  Get:1 http://security.ubuntu.com natty-security Release.gpg [198 B]
>  Get:2 http://security.ubuntu.com natty-security Release [39.8 kB]
>  Ign http://us-east-1.ec2.archive.ubuntu.com natty InRelease
>  Ign http://us-east-1.ec2.archive.ubuntu.com natty-updates InRelease
>  Ign http://us-east-1.ec2.archive.ubuntu.com natty Release.gpg
>  Ign http://us-east-1.ec2.archive.ubuntu.com natty-updates Release.gpg
>  Ign http://us-east-1.ec2.archive.ubuntu.com natty Release
>  Ign http://us-east-1.ec2.archive.ubuntu.com natty-updates Release
>  Ign http://us-east-1.ec2.archive.ubuntu.com natty/main amd64
> Packages/DiffIndex
>  Ign http://us-east-1.ec2.archive.ubuntu.com natty/universe amd64
> Packages/DiffIndex
>  Ign http://us-east-1.ec2.archive.ubuntu.com natty/main TranslationIndex
>  Ign http://us-east-1.ec2.archive.ubuntu.com natty/universe
> TranslationIndex
>  Ign http://us-east-1.ec2.archive.ubuntu.com natty-updates/main amd64
> Packages/DiffIndex
>  Ign http://us-east-1.ec2.archive.ubuntu.com natty-updates/universe amd64
> Packages/DiffIndex
>  Ign http://us-east-1.ec2.archive.ubuntu.com natty-updates/main
> TranslationIndex
>  Ign http://us-east-1.ec2.archive.ubuntu.com natty-updates/universe
> TranslationIndex
>  Err http://us-east-1.ec2.archive.ubuntu.com natty/main Sources
>    403  Forbidden [IP: 10.210.205.172 80]
>  Err http://us-east-1.ec2.archive.ubuntu.com natty/universe Sources
>    403  Forbidden [IP: 10.210.205.172 80]
>  ...
>
>
>  Trying to access a specific file manually shows that there're
> interleaving 404 and 403 responses for it (the file indeed doesn't exists).
> And generally situation look like 403 is being intermittently returned for
> any url (be it 200 or 404 one).
>
>  ubuntu@ip-10-243-34-224:~$ wget
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
>  --2012-02-14 13:42:12--
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
>  Resolving us-east-1.ec2.archive.ubuntu.com... 10.250.142.223,
> 10.252.111.96, 10.202.26.15, ...
>  Connecting to us-east-1.ec2.archive.ubuntu.com|10.250.142.223|:80...
> connected.
>  HTTP request sent, awaiting response... 404 Not Found
>  2012-02-14 13:42:12 ERROR 404: Not Found.
>
>  ubuntu@ip-10-243-34-224:~$ wget
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
>  --2012-02-14 13:42:14--
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
>  Resolving us-east-1.ec2.archive.ubuntu.com... 10.210.205.172,
> 10.250.142.223, 10.252.111.96, ...
>  Connecting to us-east-1.ec2.archive.ubuntu.com|10.210.205.172|:80...
> connected.
>  HTTP request sent, awaiting response... 403 Forbidden
>  2012-02-14 13:42:14 ERROR 403: Forbidden.
>
>  ubuntu@ip-10-243-34-224:~$ wget
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
>  --2012-02-14 13:42:16--
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
>  Resolving us-east-1.ec2.archive.ubuntu.com... 10.202.26.15,
> 10.210.205.172, 10.250.142.223, ...
>  Connecting to us-east-1.ec2.archive.ubuntu.com|10.202.26.15|:80...
> connected.
>  HTTP request sent, awaiting response... 404 Not Found
>  2012-02-14 13:42:16 ERROR 404: Not Found.
>
>  ubuntu@ip-10-243-34-224:~$ wget
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
>  --2012-02-14 13:42:17--
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
>  Resolving us-east-1.ec2.archive.ubuntu.com... 10.252.111.96,
> 10.202.26.15, 10.210.205.172, ...
>  Connecting to us-east-1.ec2.archive.ubuntu.com|10.252.111.96|:80...
> connected.
>  HTTP request sent, awaiting response... 404 Not Found
>  2012-02-14 13:42:17 ERROR 404: Not Found.
>
>  ubuntu@ip-10-243-34-224:~$ wget
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
>  --2012-02-14 13:42:18--
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
>  Resolving us-east-1.ec2.archive.ubuntu.com... 10.250.142.223,
> 10.252.111.96, 10.202.26.15, ...
>  Connecting to us-east-1.ec2.archive.ubuntu.com|10.250.142.223|:80...
> connected.
>  HTTP request sent, awaiting response... 404 Not Found
>  2012-02-14 13:42:18 ERROR 404: Not Found.
>
>  ubuntu@ip-10-243-34-224:~$ wget
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
>  --2012-02-14 13:42:19--
> http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/natty/main/source/Sources
>  Resolving us-east-1.ec2.archive.ubuntu.com... 10.210.205.172,
> 10.250.142.223, 10.252.111.96, ...
>  Connecting to us-east-1.ec2.archive.ubuntu.com|10.210.205.172|:80...
> connected.
>  HTTP request sent, awaiting response... 403 Forbidden
>  2012-02-14 13:42:19 ERROR 403: Forbidden.
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/linaro-android-infrastructure/+bug/932088/+subscriptions
>

-- 
Thanks and Regards,
Deepti