Next steps for Hadoop on Ubuntu

Registered by James Page on 2012-04-25

Rationale:

During Ubuntu precise, packages for hadoop, hbase, hive, zookeeper and pig where delivered via PPA for use in associated charms which are now avaliable in the charm store for Ubuntu 12.04.

This work needs to be continued to provide support for new versions of hadoop (and family) and to add new features to the charms avaliable in the charm store.

Goal:

Add new features to existing hadoop charms including kerberos support, hadoop-2.0 series and support for Apache Bigtop on 12.04.

Blueprint information

Status:
Not started
Approver:
Antonio Rosales
Priority:
Medium
Drafter:
Ubuntu Server
Direction:
Approved
Assignee:
James Page
Definition:
Approved
Series goal:
Proposed for quantal
Implementation:
Deferred
Milestone target:
milestone icon ubuntu-12.10

Related branches

Sprints

Whiteboard

UDS Discussion Topics:

Quick 101 on current packaging and general approach.
 - Based on binary distributions from upstream
 - Re-use of Bigtop packaging where appropriate
 - Rebuild of native components (to support 12.04 and ARM).
Alignment to next release of Apache Bigtop and potential collaboration (Ubuntu 12.04 will be the targetted release for Bigtop 0.4.0).
 - Stick with Hadoop 1.x or move to 2.x (former 0.23) branch (or both).
 - Process and technology for feeding back patches to Bigtop (not easy ATM).
 - Work to support Ubuntu 12.04 as target version for Bigtop.
Updates to stable packages for Ubuntu 12.04
 - Release/Testing process
OpenJDK7 as default Java in 12.10 - support from upstreams and testing process
Revisit packaging from source.
Testing of packages and functionality
Enable kerberos authentication in the current charmset?
Use of puppet modules in juju charms.

http://pad.ubuntu.com/uds-q-servercloud-q-bigdata-hadoop

User Stories:

Toby want todo some testing of hadoop-2.0 prior to upgrading his existing hadoop-1.0.x deployments. He is easily able to deploy a hadoop-2.0 cluster using Juju and the hadoop charm from the Juju Charm Store.

Frank likes the idea of using Juju to deploy hadoop but he's unsure about the quality of the charms. He can easily see the depth of testing being undertaken on each of the charms and decides that this is the right choice for his deployment.

Janice wants to deploy a secure hadoop cluster. She is able to leverage the hadoop charms inconjunction with kerberos to support this requirement.

Assumptions:

- hadoop-2.0 will be packaged alongside hadoop-1.0.x (different PPA or different pkg names).
- Hadoop will run OK on openjdk-7
- Suitable resources for regular testing of hadoop charms.
- Changes can be fed back to bigtop easily
- All work will also target Ubuntu 12.04 as well as 12.10
- No new releases of components/charms for Ubuntu 11.10.
- Packages will continue to support ARM based architectures.

Test Plans:

Charms will be automatically tested.

Release Note:

N/A - still outside of the main archive.

(?)

Work Items

Work items:
Update bigtop-utils and hadoop charms to support openjdk-7: DONE
Testing of existing hadoop charms with openjdk-7: DONE
Update hadoop charms to support kerberos based authentication: POSTPONED
Review use of puppet modules in hadoop charms: DONE
Write/update charms to use bigtop 0.4.0 release repositories: INPROGRESS
Setup automated testing of charms using bigtop tests: POSTPONED
Support usage of Oracle EJRE for ARM based deployements for hadoop 1.0.x: INPROGRESS

Dependency tree

* Blueprints in grey have been implemented.

This blueprint contains Public information 
Everyone can see this information.