Installation service for physical nodes deployments (UEC and more)

Work Items for natty-alpha-2:
[clint-fewbar] investigate adding mac address to debian installer's HTTP requests for preseed file: DONE
[clint-fewbar] write up proposal for Packaging and Enhancing Cobbler: DONE
[clint-fewbar] get the web interface accessible after installation: DONE
[davewalker] setup bzr import of cobbler git tree at lp:cobbler: DONE
[kirkland] push to common branch, lp:~ubuntu-virt/cobbler/ubuntu: DONE
[kirkland] upload to common PPA, ppa:ubuntu-virt/ppa: DONE
[kirkland] 2_installing - initially package cobbler for Ubuntu: DONE
[davewalker] evaluate Edison vs. Cobbler: DONE
[clint-fewbar] Work with Cobbler development to upstream patches and coordinate release: DONE
[clint-fewbar] Create cobbler-web binary package: DONE
[kirkland] get the package installing (postinst/prerm problems): DONE
[kirkland] 2_installing - get cobbler running well on Ubuntu: DONE
[kirkland] get cobbler importing Fedora ISO: DONE
[kirkland] get cobbler's wake-on-lan working in Ubuntu, r1904-r1905: DONE
[zulcss] get cobbler pxe netbooting systems: DONE
[kirkland] get cobbler deploying Fedora from Ubuntu: POSTPONED
[zulcss] get cobbler importing Ubuntu ISO: DONE
[zulcss] get cobbler deploying Ubuntu From Ubuntu: DONE
[zulcss] create proof of concept templating script for tftp/pxes/etc using cloud-init syntax: POSTPONED
[zulcss] 2_installing - get cobbler into the archive (universe): DONE
[smoser] create new cloud type for cloud-init that uses local file for metadata: POSTPONED
[clint-fewbar] testing - test installing natty and lucid from Natty: DONE
[kirkland] 1_mirroring - decide on mirror/proxy approach (squid-deb-proxy): DONE
[kirkland] 1_mirroring - get mirroring working out of the box, configured and running: DONE
[kirkland] 3_triggering - get wake-on-lan working as a trigger: DONE

Work items:
[kirkland] get cobbler deploying Fedora from Ubuntu: POSTPONED
2_installing - create or improve a web interface for triggering/driving/tracking installations: POSTPONED
2_installing - add an Ubuntu CSS scheme/skin: POSTPONED
3_triggering - get ipmi working as a trigger (best effort): POSTPONED
4_configuring - install/configure puppet master on the infrastructure server: POSTPONED
4_configuring - install/configure puppet client on target installs: POSTPONED
4_configuring - exchange puppet keys: POSTPONED
[kirkland] 4_orchestration - package mcollective 1.0.0: DONE
[kirkland] 4_orchestration - merge mcollective 1.01: DONE
[kirkland] 4_orchestration - package rabbitmq-stomp (dependency of mcollective): DONE
5_monitoring - install/configure rsyslog client/server: POSTPONED
5_monitoring - install/configure collectd client/server: POSTPONED
5_monitoring - install configure ntp client/server: POSTPONED
6_debug - support adding openssh-server and ssh-import-id via puppet on target: POSTPONED

Status: Cobbler is in Universe. Tested installing natty current from natty current, and installing lucid from natty. Bugs reported and being worked on.

Patches upstream status:

MERGED:
03_manpage-whatis-entries.patch - https://fedorahosted.org/cobbler/ticket/616
10_detect_ubuntu.patch - https://fedorahosted.org/cobbler/ticket/617

We can maintain the list of patches needing to be upstreamed here until the package has stabilized, at which piont the DEP3 headers in debian/patches should be sufficient. -clint-fewbar

Had a meeting with Mathias last week where we talked through the entire grand vision, as well as the small steps we need to take in Natty to get there.

The big pieces:
 1) mirroring
    - lots of options here, Mathias prefers apt-cacher-ng, Dustin likes approx, some other like squid-deb-proxy, apt-proxy, as well as full mirroring systems
    - this part should be pluggable, within a few constraints
    - for natty: pick one (apt-cacher-ng or approx) and get it working
      - Dustin: Dave and I talked to Michael Vogt today; sounds like squid-deb-proxy is the best way to go
 2) installing
    - cobbler seems to be the front-runner right now, though needs to be packaged for Ubuntu; see: https://fedorahosted.org/pipermail/cobbler/2010-June/005720.html
    - alternatives include uec-provisioning and fai
    - for natty: package cobbler and get it working with ubuntu
 3) triggering
    - cobbler does some of this (ipmi), might need powerwake for WoL
    - for natty: ensure that we have wake-on-lan, and possibly ipmi support
 4) configuring
    - puppetmaster as part of infrastructure host
    - puppet client on the deployed minimal system
    - puppet registration (ca/csr/token signing not yet automated, need work from upstream)
    - puppet master could inject cert information into preseed as part of the installation service
    - installation service also runs puppet ca service
    - eventually use couchdb backend for cobbler and puppet (not natty, long term though)
    - for natty: configure puppet master as part of installation service, install puppet client (and possibly exchange keys) on baremetal install
 5) monitoring
    - rsyslog and collectd in guests
    - rsyslog at cobbler/preseed to track installation logs (check what cobbler already supports)
    - add collectd post install via puppet(?), would make config easier
    - ntp server and client configuration
    - munin/nagios and hadoop -> natty + 1
    - for natty: rsyslog, collectd, ntp
 6) debugging
    - ssh, mcollective on clients for debug
    - for natty: package mcollective (maybe), add ssh-import-id support

For natty, the goal is an all-in-one proof-of-concept "Ubuntu Infrastructure Server". We'll create a new package (or just rename uec-provisioning) with a set of binary packages, one for each component that might one day be separated. These will depend or recommend other various pieces to do their part. Any new code we have to write will go here.

allocate (future) hooks for provisioning of cloud instances (ensemble, etc): TODO