R&D: rebasing, tagging + repo

[danilo, 2012-11-22] I've looked at the notes and still don't remember exactly what is this one about. "git pull --tags" not being used tells me what the implementation problem is, but I am not sure what the practical problem is.
[asac, 2013-01-08]: the practical problem is that developeres that rebase their trees usually put a tag before the rebase and its against common practices to ask them to create a head as well or instead. This causes failures to build old releases frequently and if repo would just pull tags we would eliminate many of such build failures.
[danilo, 2013-02-27] The goal is to get to an 'informational' status where we know what the next steps would be.
[pfalcon 2013-03-26] Well, it's always like that - when you need a problem case, it's not so easy to reproduce it. I spent good deal of yesterday (checking out entire android platform is slow!) semi-randomly trying old releases, and found just 2 failure cases:

repo init -u git://android.git.linaro.org/platform/manifest.git -b linaro-android-11.06-release -m LEB-panda.xml --depth=1
GitError: android/device/ti/proprietary-open update-ref: fatal: af25ad97522c7681751740b674334e0fa7b7785f^0: not a valid SHA1
repo init -u git://android.git.linaro.org/platform/manifest.git -b linaro-android-12.06-release -m tracking-panda.xml --depth=1
GitError: device/linaro/pandaboard update-ref: fatal: 5bb81146e021951417cfd93394207dda353bc0a5^0: not a valid SHA1

[pfalcon 2013-03-26] Test repo repository:
http://git.linaro.org/gitweb?p=people/pfalcon/repo-rebased/repo-rebased.git;a=summary
http://git.linaro.org/gitweb?p=people/pfalcon/repo-rebased/manifest.git
[pfalcon 2013-03-26] Test repo now has test script to check fetchability of rebased revisions which were tagged/branched before rebase. Good news that in both cases fetching by revision work! In other cases, contemporary repo appear to work as we expect, we wouldn't need to do anything about it, just use properly. That also explains why I had hard time finding broken checkout among recent releases. What's left is to analyze identified breakages to understand why *they* happen.
[pfalcon 2013-03-27] Moved detailed fetch failure analysis to https://wiki.linaro.org/PaulSokolovsky/ReleaseAudit . 2 issues above turned out to be false positives, but in 12.06 actual issue was found, confirmed the revision used doesn't have underlying tag, so failure expected.
[pfalcon 2013-03-27] Recommendations for improving the situation: We should enforce usage of underlying tags for all releases, possibly even for official daily builds (i.e. allow only personal builds to use arbitrary untagged revs). One obvious way to do that is to validate on android-build that manifest doesn't contain nay raw SHA revisions (but only branches/tags). That was already proposed before, but was opposed by Zach as time-consuming. Well, that's how everyone else does that (including AOSP upstream), and adopting this scheme (and associated changes) may improve Linaro Android release process. Alternatively, we can do it upside-down - try to verify that each raw rev has underlying tag. That should be roughly possible, but I wary of possible false positives/negatives with such approach (i.e. it will be harder to debug and prove that it works as expected). If we want to go real pro, we also probably should re-audit all previous release and clearly mark those which are no longer fetchable/reproducible. Unfortunately, this is going to be very time and traffic consuming operation - it appears there's no easy way to check if git repo has a revision without fetching it first. My hope was on gitweb's commit search capabilities, but I found it works not reliable and gives weird results sometimes.

Prior art timeline:
2011-08-30 - Our patch repo to to fetch all tags submitted to https://review.source.android.com/25843 (no longer accessible)
2011-09-05 - https://wiki.linaro.org/Platform/Android/PatchedRepo created
2011-09-06 - https://wiki.linaro.org/PaulSokolovsky/Release1108Audit created (now renamed to ReleaseAudit)
2011-09-xx - kernel.org together with all AOSP infra goes down for more than a month
2011-09-xx - Further testing on #25843 by ourselves shows that it leads to additional performance issues (with our strained repo-based mirror at least (which since then was replaced by seeded builds))
2011-11-16 - Comment on #25843 that https://gerrit-review.googlesource.com/#/c/25764/ implemented needed functionality
2011-11-29 - #25843 abandoned

Meta:
Headline: Find how we can get to working repo sync with git repositories which are frequently rebased.
Acceptance: There is an implementation plan defined for how to get to fully usable pinned manifests which work even with git repositories which are frequently rebased.

Work items:
Recover previous timeline dealing with this issue: DONE
Identify recent usecase(s) of checkout failures due to invalid revs: DONE
Analyze if revs above has underlying tags: DONE
Prepare tagging/rebasing small test repo: DONE
Add test script which test behavior for different cases of rebasing on test repo: DONE