Enable lightweight HTTP access for android.git.linaro.org

Registered by Данило Шеган on 2012-09-24

For a while now, our automated build service on ci.linaro.org has been using lightweight ("non-smart") HTTP access to clone off git.linaro.org. We should make that happen for android.git.linaro.org and android-build.linaro.org as well.

Blueprint information

Данило Шеган
Данило Шеган
Данило Шеган
Series goal:
Milestone target:
milestone icon 2012.11
Started by
Данило Шеган on 2012-09-24
Completed by
Данило Шеган on 2012-11-29


Headline: Allow non-smart HTTP access on android.git.linaro.org and use that for android-build.linaro.org builds where possible.
Acceptance: There is http://android.git.linaro.org/git-ro/ serving all the repositories through "non-smart" HTTP (i.e. apache serving pack files directly) and all the official manifest files are updated to use those links as well.

[2012-09-24 danilo] We might even want to go and rewrite manifest files for publishing, since significant load is coming from people doing their first-time builds by following our "rebuild" instructions. We should look for git://android.git.linaro.org and http://android.git.linaro.org/git/ URLs and replace them with new ones.
[2012-09-27 danilo] Set up test builds on https://android-build.linaro.org/builds/~danilo/panda-httpgit-seed/ and https://android-build.linaro.org/builds/~danilo/panda-httpgit-noseed/
[2012-10-26 danilo] First build gone through, but still moving this to 2012.11 (as per postmortem call).
[2012-11-13 danilo] Builds working with custom build scripts which rewrite the manifests: lp:~danilo/linaro-android-build-tools/no-seed
[2012-11-13 danilo] Seeded builds (2.5h) are still much faster than non-seeded (last one took 3.5h), though a proxy should solve that.
[2012-11-22 danilo] Setting up android.git.linaro.org to serve git repos on /git-ro with and without ".git" is more complicated than expected (https://rt.linaro.org/Ticket/Display.html?id=664): will go with a workaround for this release.
[2012-11-27 danilo] rt#664 has been resolved, we've got support for URL rewriting (it's not perfect: apache directory index rendering makes it go to the internal URL /git-http), but at least it works. Uncompressing entire pack files does not help with memory usage in smart HTTP, or progress bars with non-smart HTTP git access, and results in huge disk space usage (eg. 870MB kernel repo ended up being 34GB when kept uncompressed).
[2012-11-29 danilo] First builds using http_proxy at ci.linaro.org (now configured to keep a cache of 30GB) have been done. repo sync time is down from 41min with squid cache unpopulated (https://android-build.linaro.org/jenkins/job/danilo_galaxynexus-jb-gcc47-aosp-blob/3) to 28min with cache populated (https://android-build.linaro.org/jenkins/job/danilo_galaxynexus-jb-gcc47-aosp-blob/4). Compare this to 44 mins with seed and no proxy (https://android-build.linaro.org/jenkins/job/danilo_galaxynexus-jb-gcc47-aosp-blob/1), which doesn't include the seed download time (getting 16GB over is noticeable as well).


Work Items

Work items:
Set up lightweight HTTP /git-ro serving under android.git.linaro.org (rt#608): DONE
Set up gerrit hooks for running git update-server-info (rt #657): DONE
Compare how android builds using lightweight HTTP git access work with and without the seed: DONE
Experiment with pack size for better git pull/clone output over lightweight http: DONE
Switch to use http://android.git.linaro.org/git-ro/ and http://git.linaro.org/git-ro/ as base git URLs for android builds: DONE
Ensure all manifest files we publish are making use of http://android.git.linaro.org/git-ro repository URLs (even if they use git://android.git.linaro.org or http://android.git.linaro.org/git URLs): POSTPONED
Set up http proxy on android-build.linaro.org: DONE

This blueprint contains Public information 
Everyone can see this information.


No subscribers.