Provide CI setup for Openembedded Engineering Builds

Meta:
Headline: OpenEmbedded CI builds now use persistent builders, which drastically cuts typical build time.
Acceptance: Persisten build slave type with underlying custom AMI set up, OE builds switched to use it.

[2012-11-22 danilo] I find a few things unclear:
  Do we care what Ubuntu release we use as the base (if not, I'd suggest 12.04)?
  Can we get the details on how the image needs to be set up (we want to ensure we can recreate them and understand exactly the steps, because we are in the business of building general infrastructure :)?
  I, specifically, don't like the approach where hrw ssh's in and sets it up: I want the job to be automated, potentially using custom AMI creation scripts we've got in lp:linaro-aws-tools.
[fboudra, 2012-11-22] +1 for Ubuntu 12.04, +1 for the custom AMI approach, ideally we provide a script to set up automatically the slave.
[hrw, 2012-11-22] Let me add some information then:
- 12.04 is great.
- We do not need ssh access - standard scripts already take care of everything needed from build side.
- There will be a need for some remote storage mounted on instance in a way that 'ubuntu' user can read/write there. We will keep sources and sstate-cache files there to speed up builds.
- sources, patches used/generated by build can be generated on such storage as well for publishing (to comply with licenses)
- would be great to have rsync job which will update http://snapshots.linaro.org/openembedded/sources/ after each job
[fboudra, 2012-11-22] wrt rsync, we have a job to publish sources on snapshots. what's wrong with it?
[hrw, 2012-11-22] We have several GBs of sources. Now we are sending the same files over and over. Soon we will change to freeze revisions of OE layers so we will build ~same daily.

[hrw 2013-01-28] We need persistant instances with shared storage. This way each build can use results of previous builds (but each will start with clean TMPDIR) resulting in builds lasting minutes instead of hours (but such ones will also happen).
[pfalcon 2013-02-11] hrw: Even we don't have shared storage, but have "farm" of ~3 persistent slaves they should provide pretty good performance - they won't be synced, but will converge into having most of stuff cached for all builds (assuming we can share cached stuff among different types of builds)
[pfalcon 2013-02-11] It seems I'm taking this for this cycle.
[hrw 2013-02-11] Paul: I owe you beer in HK.
[pfalcon 2013-02-12] We have test builds working as expected (wrt to slave usage) with https://ci.linaro.org/jenkins/job/openembedded-armv8-minimal , builds #137 and on.
[pfalcon 2013-02-13] Script to create AMI for persistent builders is captured as http://bazaar.launchpad.net/~linaro-aws-devs/linaro-aws-tools/trunk/view/head:/linaro-ami/create-oe-persistent-ami.sh
[pfalcon 2013-02-26] Over last couple of weeks we with Marcin debugged few issues and did some tweaks, I guess persistent slave support can be cast done.
[dzin, 2013-02-26] please add roadmap, headline and acceptance
[pfalcon] Re: "Mirror OpenEmbedded git repositories" WI, quoting Marcin: "with persistent slaves there is much less need for those mirrors as
we do just 'git pull' to the latest instead of full cloning of OE repos."

Work items:
Setup EC2 slave instance for doing OE builds: DONE
Prepare build slave AMI with persistent EBS storage: DONE
Setup ci.linaro.org to use above instance and make sure instance is stopped (not terminated) on completion: DONE
Test persistent job: DONE
Migrate existing OE jobs to persistent setup: DONE
Mirror OpenEmbedded git repositories: POSTPONED