Publish Android Builds artifacts to another host (e.g. snapshots.linaro.org)

Notes:
[pfalcon 2011-12-08]
Basic approach would be to find suitable Jenkins publisher plugin, and verify that it satisfies our requirements: pushes artifacts to another host at the end of build; doesn't leave them on android-build; has secure configuration and workflow; ideally, updates artifact links in Jenkins to point directly to new host.
[asac 2011-12-09]: input: the hierarchy of the builds should be like: snapshots.linaro.org/android/~BUILD_USER/BUILD_NAME/NUMBER/... so snapshots.linaro.org/android/~linaro-android/staging-origen ... or .../~asac/staging-origen
[asac 2011-12-09]: updated the acceptance criteria to make clear that we for now mostly care about snapshots.linaro.org publishing (and more hosts is on top), but that _all_ builds should go there for now. also updated requirements that build frontend shows the proper links.
[pfalcon 2011-12-09] There're basicly three candidate solutions: SCP Plugin, Publish Over SSH Plugin (this said to use SFTP), and adhoc command in build script. Plugins considered preferred, because there's hope that they can replace artifact links in Jenkins. Unfortunately, so far there're no indication that to be actually true (i.e. they are likely *copy* artifacts somewhere else, not move). Latest version of SCP Plugin supports key auth.
[danilo 2011-12-09] We also need to update android-frontend to point to new links. Not sure how we can preserve compatibility with older builds currently hosted on android-build.l.o.
[pfalcon 2011-12-09] 2 danilo: yes, and that's not the only problem, there's also details of LAVA integration, etc. I perplexed with desire to move all builds elsewhere - we want to change too many things at once, and at very short period of time (2 weeks). IMHO, we'd rather concentrate on task at hand - have *some* builds protected by click-thru, and then next milestone assess if that works well enough to move all builds that way (will write more in mail on Mon).
[pfalcon 2011-12-12] Review of Jenkins SCP approaches: https://wiki.linaro.org/PaulSokolovsky/JenkinSCPNotes
Discussed with Danilo, we're ok with doing snapshots.l.o switchover incrementally during this milestone, but still target complete switchover for this BP, otherwise will need to maintain 2 systems (and have much more complicated security audit)
[pfalcon 2011-12-12] In Jenkins SCP plugin, key path should be absolute FS one. Jenkins ticket which led to addition of key auth: https://issues.jenkins-ci.org/browse/JENKINS-1269
[dooferlad 2011-12-12] We should be able to get IT to set up a chrooted user that Jenkins can upload to (http://www.techrepublic.com/blog/opensource/chroot-users-with-openssh-an-easier-way-to-confine-users-to-their-home-directories/229) and mount /srv3/snapshots.linaro.org/ into that user's home directory. We then just need one of those Jenkins plugins to do what we need...
[dooferlad 2011-12-12] https://rt.linaro.org/Ticket/Display.html?id=226
[pfalcon 2011-12-14] In the end, we're using "Publish over SSH/SFTP" plugin, because IS set up SFTP-only account, and actually this plugin provides many more features and flexibility than SCP plugin.
[pfalcon 2011-12-14] https://rt.linaro.org/Ticket/Display.html?id=232
[pfalcon 2011-12-19] "Update existing jobs using jobs-mangle" - vetoed by Zach until Wed
[pfalcon 2011-12-22] Backbone of the new infrastructure launched and works well. Some tasks didn't fit into this miestone, marked as POSTPONED so far. They can be BUGREPORTED, or probably better to spawn a new BP with them to keep them in focus.
[pfalcon 2011-12-23] POSTPONED items split to https://blueprints.launchpad.net/linaro-android-infrastructure/+spec/linaro-android-snaphosts-publish-finish

Meta:
Roadmap id: ANDROID2011-ACCEL-OVERLAYS
Headline: Android build artifacts are now published to snapshots.linaro.org, which allows us to publish accelerated builds for boards which require license acceptance before download.
Acceptance: All jobs on Android Build artifacts are published to snapshots.linaro.org (with option to publish to _more_ hosts later) and at least the snapshots.linaro.org publishing works properly. Published artifacts are not available on Android Build. The Android Build UI however, still shows the artifact links, just pointing to the snapshots.linaro.org location.

Work items:
[pfalcon] Research Jenkins publisher plugins: DONE
[pfalcon] Test selected plugins: DONE
[pfalcon] Add "build copycat" feature (don't build anything, copy from existing build to ease debugging): DONE
Confirm publishing host: DONE
[dooferlad] Set up on publishing host infrastructure and credentials for remote access: DONE
[pfalcon] Install plugin on Android Build: DONE
[danilo] Cleanup Jenkins paths when pushing since it uses username_buildid/build_number: DONE
[pfalcon] Setup test publishing jobs: DONE
Confirm which jobs need to be remote-published: DONE
Update "blank" (template) job: DONE
Update existing jobs: DONE
[pfalcon] Reorder artifact publish vs LAVA request steps to avoid races: DONE
[pfalcon] Update existing jobs using jobs-mangle: DONE
[pfalcon] Update LAVA integration to use new URLs: DONE
[pfalcon] Update frontend link to download area (easy): DONE
Do not publish build results on current URLs on android-build.l.o: POSTPONED
Update frontend links to individual artifacts (more involved): POSTPONED
[dooferlad] Update frontend LAVA integration (maybe even more involved): POSTPONED
Migrate old builds archive to snapshots.l.o: POSTPONED
Production configuration and launch: DONE