Preparing CEPH for main inclusion

Registered by Clint Byrum on 2011-10-11

CEPH has become a first class object and block storage mechanism, and has integration hooks for key pieces of Ubuntu Cloud Infrastructure. Especially interesting is CEPH-RBD for storing kvm block devices on the scalable CEPH block store, and the Hadoop HDFS shim to replace HDFS with CEPH in hadoop clusters.

Blueprint information

Antonio Rosales
Ubuntu Server Team
Clint Byrum
Series goal:
Accepted for precise
Milestone target:
milestone icon ubuntu-12.04-beta-1
Started by
Robbie Williamson on 2012-02-17
Completed by
Clint Byrum on 2012-04-16

Related branches



Work Items:
[canonical-kernel-team] investigate large xattr support patch from Ted T'so in ext4: POSTPONED
[canonical-kernel-team] potentially help push large xattr support to ext4: POSTPONED
confer with kernel-team to decide on which to recommend, ext4/xfs/btrfs (xfs or btrfs, ext4 will not work due to large xattr problems): DONE
define Ubuntu supported configurations of CEPH (only librbd/librados made it into main for precise): DONE
improve ceph charms: POSTPONED
integrate ceph charm with openstack charms: POSTPONED
libvirt changes needed: POSTPONED
MIR ceph ( ): DONE
[serge-hallyn] enable rbd and rados in qemu-kvm builds: DONE
[nijaba] correctly message technology preview for non-supported bits: DONE
investigate apache + 100 continue support for rados S3 GW: POSTPONED
push nagios and collectd plugins into Ubuntu/Debian/Upstream: POSTPONED
setup automated build recipe against upstream source: POSTPONED
import upstream test suite into archive: POSTPONED
[sconklin] ext4/btrfs/xfs testing in QA: POSTPONED

qemu-kvm librados builds are blocked on bug 932898 being fix released.

No formal spec needed IMO, this is just a pile of straight forward actions.

 * gceph is not reasonable to be in main (upstream doesnt want to support)

    Today rbd is loaded loaded at runtime

 - MIR librbd and librados for qemu/kvm
  - compile qemu-kvm with ceph (rbd, rados) block device support, see:
 - MIR all of ceph for hosting ceph
 - include latest libvirt rbd support (for rbd authentication)
 - Existing Openstack Support
   - Nova since cactus can use CEPH for block
   - Diablo added glance support for CEPH for image storage
 - Dreamhost using mainline BTRFS and debian kernel for current deployment
- Dreamhost launching at scale in a few "weeks"
- CEPH is an independent business unit under Dreamhost
- Dreamhost using fastcgi + apache for rados gateway (can use nginx, too)

- Upstream makes a release every 2 weeks
- No stable release branches upstream
- Limited deployment outside Dreamhost
- unencrypted over the wire
  - apps can implement security on top of it
  - IPv6 is supported
Additional Opportunities:
- Integrate tightly with OpenStack in packaging
- Add support for ceph in juju openstack charms
- BTRFS ideal, XFS also working well, ext4 missing "large xattr" support
- Upstream asserts that CEPH is under heavy development (cmdline programs remind user at every execution)
 - in the openstack use case, for kvm, there is no need for kernel code at all
  - if we want to support xen, it would require using the block device kernel code for rdb
Support Targets:


Work Items

Dependency tree

* Blueprints in grey have been implemented.