Evaluation of self-sufficient git hosting on git.linaro.org

Registered by Данило Шеган on 2012-06-08

Our current code-hosting set-up spans several systems (git.linaro.org, android.git.linaro.org, github and Launchpad). We should aim to get to at most two systems (one for bazaar, another for git) that provide all the features we need and desire (or at least the most important ones).

This has been discussed during Linaro Connect in May/June 2012. Notes from the session are available on
  http://summit.linaro.org/lcq2-12/meeting/20739/code-hosting/

Blueprint information

Status:
Complete
Approver:
Данило Шеган
Priority:
High
Drafter:
Milo Casagrande
Direction:
Approved
Assignee:
Georgy Redkozubov
Definition:
Approved
Series goal:
Accepted for trunk
Implementation:
Informational Informational
Milestone target:
milestone icon 2012.11
Started by
Данило Шеган on 2012-07-10
Completed by
Georgy Redkozubov on 2012-11-29

Related branches

Sprints

Whiteboard

[danilo 2012-06-08] Setting the priority to 'essential' before the planning call for 2012.06 (so it flows to the top of the backlog list for re-prioritization).
[stevanr 2012-06-12] Creating evaluation spreadsheet for aforementioned code hosting systems.
[stevanr 2012-06-13] Evaluation spreadsheet on https://docs.google.com/a/linaro.org/spreadsheet/ccc?key=0Amowg117_qJzdEYwT25Nd3gwRzVmY2hpb0N2V2xpVlE#gid=0
[danilo 2012-06-22] Start for 2012.07
[danilo 2012-07-10] Move back to backlog, no resources to start this cycle.
[danilo 2012-07-20] Scheduled.
[dooferlad 2012-08-06] Github enterprise is designed for internal use only, so not suitable.
[asac 2012-8-13] what's the status on this one? where are we standing? what's next, how can this move on?
[dooferlad 2012-8-13] The spreadsheet is just about complete (I thought we were closer, but just got told some information was missing). From the information we have it looks like one of the commercial offerings will fit with github possibly winning.
[stevanr 2012-08-14] We now have a pretty decent looking (and populated) evaluation doc, I think the next step should be to organize a session and discuss the document for a bit, thus rendering a final decision.
[dooferlad 2012-08-20] Just added word from github about private projects. Personally I would go with one of the commercial offerings to offset us doing sys admin work. In terms of response time gitorius won hands down. In terms of privacy gitorius also win because they will set up a server with a mix of public and private repos for us, while github requires us to host on their servers (which some customers may not like). Gitorius is also entirely open source. Of course there is the issue of them having some access to the host to do admin.
[dzin 2012-08-21] At risk for delivery for 12.08.
[danilo 2012-08-24] Scott, Asac, Joey, Loic, Zach, all tech leads, start a discussion asap, target to 2012.10.
[gesha, 2012-09-26] Pushed linux-linaro-tracking to https://github.com/Linaro/test-kernel
[gesha, 2012-09-26] Test job is setup https://ci.linaro.org/jenkins/job/gesha-github-test_beagle-omap2plus
[stevanr 2012-09-26] Updated the spreadsheet with rhodecode.org code hosting system: https://docs.google.com/a/linaro.org/spreadsheet/ccc?key=0Amowg117_qJzdEYwT25Nd3gwRzVmY2hpb0N2V2xpVlE#gid=0
[gesha, 2012-10-01] Successfull kernel build from public repo on github: https://ci.linaro.org/jenkins/job/gesha-github-public-test_beagle-omap2plus/
[gesha, 2012-10-01] Kernel build from private repo on gihub: https://ci.linaro.org/jenkins/job/gesha-github-test_beagle-omap2plus/
There are several successfull jobs - #20, #22, #23 and one failed with "fatal: git upload-pack: aborting due to possible repository corruption on the remote side." error - #21 which shows that same problems as we have in Linaro can pop up on Github.
[gesha, 2012-10-02] Github doesn't support subdirectories.
[stevanr 2012-10-03] Rhodecode pilot: http://ec2-23-23-30-175.compute-1.amazonaws.com/
[stevanr 2012-10-07] After some live testing and evaluating Rhodecode on AWS instance, it proved to work very slow with large repositories like ours (both git server and web get blocked) so we're back to github.
[dooferlad 2012-10-08] Any thoughts on http://gitorious.com/ vs github?
[gesha 2012-10-09] SUbdirectories is corner stone for android git layout. Talked to github support about subdirectories. Email thread https://pastebin.linaro.org/995/
[stevanr 2012-10-16] An email from Marcin Kuzminski about the RhodeCode scalability: https://pastebin.linaro.org/1008/
[danilo 2012-10-26] Moved to 2012.11 in a post-mortem: changed it to evaluation so BP gets finished soon.
[gesha 2012-11-07] Rhodecode demo instance is running on https://ec2-54-242-196-100.compute-1.amazonaws.com
[gesha 2012-11-07] http://ec2-54-242-196-100.compute-1.amazonaws.com is used for git-ro access
[gesha 2012-11-13] Android build successfully pulling from git server using "dumb" http protocol https://android-build.linaro.org/jenkins/job/gesha_rhodecode-test/29/consoleFull
[gesha 2012-11-20] Performance was improved by switching to beta version, which has lightweight dashboard option
[gesha 2012-11-20] Next steps will be deciding how to proceed with gerrit and dumb http access to share/use the same Rhodecode ACLs
[gesha 2012-11-26] Issues, possible workarounds, notes on Rhodecode evaluation use https://docs.google.com/a/linaro.org/document/d/1JKaITKRBfXhA-x-Z8HF4HMinRdJP64PYtjvJcdPK0lo/edit
[gesha 2012-11-26] Rough migration plan for the git.linaro.org to Rhodecode server https://docs.google.com/a/linaro.org/document/d/1uxvn10SXqMtYE_0KvY6AWfkadU2_ZdJCChzjjAEbA_k/edit
[danilo 2012-11-29] Modify the acceptance to match what we agreed last month this blueprint should be: evaluation and decision.

Meta:
Headline: Provide a git hosting which requires little or no intervention from the IS/Infrastructure team for setting up git repositories, with proper account/privacy management through the web UI.
Acceptance: Have a proposed implementation plan and decision for git code hosting on git.linaro.org. Potentially have a plan for android.git.linaro.org migration as well.
Roadmap id: CARD-148

(?)

Work Items

Work items for 2012.06:
[stevanr] Evaluate gitorious, gitolite, cgit, gitweb, gitlab: DONE

Work items for 2012.08:
[dooferlad] Add github (hosted service and enterprise version [runs on our servers]) to the comparison: DONE
[dooferlad] Re-evaluate everything with new items added to the spreadsheet: DONE

Work items for 2012.09:
[danilo] Start up a discussion among tech leads to choose the most appropriate solution: DONE
[danilo] Update the acceptance/headline/workitems to match up with new requirements for 2012.10: DONE

Work items for 2012.10:
[gesha] Import linux-linaro-tracking git repository to github.com/Linaro: DONE
[stevanr] Evaluate rhodecode.org: DONE
[gesha] Set-up a CI kernel build pulling kernel from github: DONE
[gesha] Email github to see if they are interested in us hosting our code there, if there're further things they could do for us: DONE

Work items for 2012.11:
[gesha] Set up bigger rhodecode instance and host android git on it: DONE
[gesha] Pilot project — android builds on rhodecode instance: DONE
[gesha] Setup gerrit on the rhodecode demo ec2 instance: DONE
[gesha] Try to get lightweight HTTP git access working: DONE
[gesha] Share the findings and decide whether we want to go with github or continue to self-host: INPROGRESS

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.