Publish Android Builds artifacts to another host (e.g. snapshots.linaro.org)

Registered by Paul Sokolovsky on 2011-12-08

To accommodate need for click-thru licenses and other requirements from vendors, we want to push built artifacts to another host, where we can easily apply needed separation and access controls. Such artifacts should not be accessible on Android Build by any means.

Blueprint information

Status:
Complete
Approver:
Данило Шеган
Priority:
Essential
Drafter:
Paul Sokolovsky
Direction:
Approved
Assignee:
Paul Sokolovsky
Definition:
Approved
Series goal:
Accepted for trunk
Implementation:
Implemented
Milestone target:
milestone icon 11.12
Started by
Paul Sokolovsky on 2011-12-09
Completed by
David Zinman on 2011-12-23

Related branches

Sprints

Whiteboard

Notes:
[pfalcon 2011-12-08]
Basic approach would be to find suitable Jenkins publisher plugin, and verify that it satisfies our requirements: pushes artifacts to another host at the end of build; doesn't leave them on android-build; has secure configuration and workflow; ideally, updates artifact links in Jenkins to point directly to new host.
[asac 2011-12-09]: input: the hierarchy of the builds should be like: snapshots.linaro.org/android/~BUILD_USER/BUILD_NAME/NUMBER/... so snapshots.linaro.org/android/~linaro-android/staging-origen ... or .../~asac/staging-origen
[asac 2011-12-09]: updated the acceptance criteria to make clear that we for now mostly care about snapshots.linaro.org publishing (and more hosts is on top), but that _all_ builds should go there for now. also updated requirements that build frontend shows the proper links.
[pfalcon 2011-12-09] There're basicly three candidate solutions: SCP Plugin, Publish Over SSH Plugin (this said to use SFTP), and adhoc command in build script. Plugins considered preferred, because there's hope that they can replace artifact links in Jenkins. Unfortunately, so far there're no indication that to be actually true (i.e. they are likely *copy* artifacts somewhere else, not move). Latest version of SCP Plugin supports key auth.
[danilo 2011-12-09] We also need to update android-frontend to point to new links. Not sure how we can preserve compatibility with older builds currently hosted on android-build.l.o.
[pfalcon 2011-12-09] 2 danilo: yes, and that's not the only problem, there's also details of LAVA integration, etc. I perplexed with desire to move all builds elsewhere - we want to change too many things at once, and at very short period of time (2 weeks). IMHO, we'd rather concentrate on task at hand - have *some* builds protected by click-thru, and then next milestone assess if that works well enough to move all builds that way (will write more in mail on Mon).
[pfalcon 2011-12-12] Review of Jenkins SCP approaches: https://wiki.linaro.org/PaulSokolovsky/JenkinSCPNotes
Discussed with Danilo, we're ok with doing snapshots.l.o switchover incrementally during this milestone, but still target complete switchover for this BP, otherwise will need to maintain 2 systems (and have much more complicated security audit)
[pfalcon 2011-12-12] In Jenkins SCP plugin, key path should be absolute FS one. Jenkins ticket which led to addition of key auth: https://issues.jenkins-ci.org/browse/JENKINS-1269
[dooferlad 2011-12-12] We should be able to get IT to set up a chrooted user that Jenkins can upload to (http://www.techrepublic.com/blog/opensource/chroot-users-with-openssh-an-easier-way-to-confine-users-to-their-home-directories/229) and mount /srv3/snapshots.linaro.org/ into that user's home directory. We then just need one of those Jenkins plugins to do what we need...
[dooferlad 2011-12-12] https://rt.linaro.org/Ticket/Display.html?id=226
[pfalcon 2011-12-14] In the end, we're using "Publish over SSH/SFTP" plugin, because IS set up SFTP-only account, and actually this plugin provides many more features and flexibility than SCP plugin.
[pfalcon 2011-12-14] https://rt.linaro.org/Ticket/Display.html?id=232
[pfalcon 2011-12-19] "Update existing jobs using jobs-mangle" - vetoed by Zach until Wed
[pfalcon 2011-12-22] Backbone of the new infrastructure launched and works well. Some tasks didn't fit into this miestone, marked as POSTPONED so far. They can be BUGREPORTED, or probably better to spawn a new BP with them to keep them in focus.
[pfalcon 2011-12-23] POSTPONED items split to https://blueprints.launchpad.net/linaro-android-infrastructure/+spec/linaro-android-snaphosts-publish-finish

Meta:
Roadmap id: ANDROID2011-ACCEL-OVERLAYS
Headline: Android build artifacts are now published to snapshots.linaro.org, which allows us to publish accelerated builds for boards which require license acceptance before download.
Acceptance: All jobs on Android Build artifacts are published to snapshots.linaro.org (with option to publish to _more_ hosts later) and at least the snapshots.linaro.org publishing works properly. Published artifacts are not available on Android Build. The Android Build UI however, still shows the artifact links, just pointing to the snapshots.linaro.org location.

(?)

Work Items

Work items:
[pfalcon] Research Jenkins publisher plugins: DONE
[pfalcon] Test selected plugins: DONE
[pfalcon] Add "build copycat" feature (don't build anything, copy from existing build to ease debugging): DONE
Confirm publishing host: DONE
[dooferlad] Set up on publishing host infrastructure and credentials for remote access: DONE
[pfalcon] Install plugin on Android Build: DONE
[danilo] Cleanup Jenkins paths when pushing since it uses username_buildid/build_number: DONE
[pfalcon] Setup test publishing jobs: DONE
Confirm which jobs need to be remote-published: DONE
Update "blank" (template) job: DONE
Update existing jobs: DONE
[pfalcon] Reorder artifact publish vs LAVA request steps to avoid races: DONE
[pfalcon] Update existing jobs using jobs-mangle: DONE
[pfalcon] Update LAVA integration to use new URLs: DONE
[pfalcon] Update frontend link to download area (easy): DONE
Do not publish build results on current URLs on android-build.l.o: POSTPONED
Update frontend links to individual artifacts (more involved): POSTPONED
[dooferlad] Update frontend LAVA integration (maybe even more involved): POSTPONED
Migrate old builds archive to snapshots.l.o: POSTPONED
Production configuration and launch: DONE

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.