Ubuntu

Move EC2 mirrors to S3

Registered by Ben Howard

Instead of using hosts to front the EC2 mirrors, we should move our mirrors to being hosted on S3. There are numerous advantages to doing so -- most notably speed, lower costs, lower maintenance costs, access policies, etc. Using S3 also realizes some cost-savings for Canonical since all intra-zone transfers are free.

Summary:
1. New mirrors would be setup in each region
2. Scripts would need to be updated to use regional mirrors
3. DNS names would need to be updated

Blueprint information

Status:
Complete
Approver:
Antonio Rosales
Priority:
High
Drafter:
None
Direction:
Needs approval
Assignee:
None
Definition:
Approved
Series goal:
Accepted for precise
Implementation:
Implemented
Milestone target:
milestone icon ubuntu-12.04-beta-1
Started by
Robbie Williamson
Completed by
Robbie Williamson

Related branches

Sprints

Whiteboard

STATUS: Due to bug 948461, and S3 handling HTTP pipelining properly, this has been postponed pending a resolution from Amazon or further discussion.

Rationale:
S3 is a faster, more scalable technology that allows us to reduce costs and increase availability for EC2 users.

Assumption:
  * Running on-EC2 mirrors is expensive and presents availability challenges.
  * on-EC2 mirrors have limited bandwidth and thus speeds can be affected by load
  * S3 has very high availability
  * S3 intra-region bandwidth is free
  * on-EC2 mirrors results in bandwidth charges when users use a mirror outside the availability zone
  * S3 is extremely fast. Generally speeds to S3 access are near-native
  * Amazon has asked for us to host our mirrors on S3
  * Uploads to S3 are free

___

Example of access via S3.
deb http://BUCKET/ubuntu/ oneiric main universe multiverse restricted
deb http://BUCKET/ubuntu/ oneiric-updates main universe multiverse restricted
deb http://BUCKET/security oneiric-security main universe multiverse restricted

deb-src http://BUCKET/ubuntu/ oneiric main universe multiverse restricted
deb-src http://BUCKET/ubuntu/ oneiric-updates main universe multiverse restricted
deb-src http://BUCKET/security oneiric-security main universe multiverse restricted

___

Decision: Build S3 Regional mirror

Work Items

[utlemming]: Add APT Meta-parsing to script: DONE
[utlemming]: Make script atomic: DONE
[utlemming]: Document process: DONE
[utlemming]: Deploy Solution: DONE
[canonical-sysadmins]: Add EC2 CNAMES: POSTPONED

    - a method of pushing pristine repositories up

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.

Nearby

This blueprint contains Public information 
Everyone can see this information.