Move EC2 mirrors to S3

Registered by Ben Howard on 2011-08-16

Instead of using hosts to front the EC2 mirrors, we should move our mirrors to being hosted on S3. There are numerous advantages to doing so -- most notably speed, lower costs, lower maintenance costs, access policies, etc. Using S3 also realizes some cost-savings for Canonical since all intra-zone transfers are free.

Summary:
1. New mirrors would be setup in each region
2. Scripts would need to be updated to use regional mirrors
3. DNS names would need to be updated

Blueprint information

Status:
Complete
Approver:
Antonio Rosales
Priority:
High
Drafter:
None
Direction:
Needs approval
Assignee:
Ben Howard
Definition:
Approved
Series goal:
Accepted for precise
Implementation:
Implemented
Milestone target:
milestone icon ubuntu-12.04-beta-1
Started by
Robbie Williamson on 2012-04-26
Completed by
Robbie Williamson on 2012-04-26

Related branches

Sprints

Whiteboard

STATUS: Due to bug 948461, and S3 handling HTTP pipelining properly, this has been postponed pending a resolution from Amazon or further discussion.

Rationale:
S3 is a faster, more scalable technology that allows us to reduce costs and increase availability for EC2 users.

Assumption:
  * Running on-EC2 mirrors is expensive and presents availability challenges.
  * on-EC2 mirrors have limited bandwidth and thus speeds can be affected by load
  * S3 has very high availability
  * S3 intra-region bandwidth is free
  * on-EC2 mirrors results in bandwidth charges when users use a mirror outside the availability zone
  * S3 is extremely fast. Generally speeds to S3 access are near-native
  * Amazon has asked for us to host our mirrors on S3
  * Uploads to S3 are free

___

Example of access via S3.
deb http://BUCKET/ubuntu/ oneiric main universe multiverse restricted
deb http://BUCKET/ubuntu/ oneiric-updates main universe multiverse restricted
deb http://BUCKET/security oneiric-security main universe multiverse restricted

deb-src http://BUCKET/ubuntu/ oneiric main universe multiverse restricted
deb-src http://BUCKET/ubuntu/ oneiric-updates main universe multiverse restricted
deb-src http://BUCKET/security oneiric-security main universe multiverse restricted

___

Decision: Build S3 Regional mirror

Work Items

[utlemming]: Add APT Meta-parsing to script: DONE
[utlemming]: Make script atomic: DONE
[utlemming]: Document process: DONE
[utlemming]: Deploy Solution: DONE
[canonical-sysadmins]: Add EC2 CNAMES: POSTPONED

    - a method of pushing pristine repositories up

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.