[MIR] ceph

Bug #932898 reported by Clint Byrum
30
This bug affects 4 people
Affects Status Importance Assigned to Milestone
ceph (Ubuntu)
Fix Released
Undecided
Clint Byrum

Bug Description

= Intro =

This is a Main Inclusion Report for CEPH.

http://ceph.newdream.net/

The request is to include the ceph source package, and all of its binaries except the following:

ceph-mds
ceph-mds-dbg
ceph-fs-common
ceph-fs-common-dbg
libcephfs
ceph-fuse

The exceptions are to separate out the CEPH POSIX filesystem, which is still considered somewhat experimental. The block and object storage underneath it, however, should be extremely stable and fully supported.

== Availability ==

CEPH has been available in Ubuntu since 10.10

== Rationale ==

CEPH is available as a drop in replacement for SWIFT in OpenStack environments, and also can be used as a highly scalable remote block store for qemu-kvm. The support for that use case is well integrated with OpenStack and helps to improve our story for Ubuntu Cloud Infrastructure. The full rationale was discussed at UDS and captured in this blueprint:

https://blueprints.launchpad.net/ubuntu/+spec/servercloud-p-ceph

Note that upstream was present and would agree that the filesystem of CEPH is not as supportable as the object and block storage system, so the two will be kept separate in the packaging and the binaries for the filesystem left out of main.

== Security ==

CEPH has not had any known CVE's reported against it.

The CEPH filesystem has had a detailed paper written describing a proposed security model:

http://www.ssrc.ucsc.edu/Papers/leung-storagess06.pdf

However, it was not implemented. This is not entirely relevant to this MIR, as the filesystem binaries will be kept out of main for now.

The lower level encryption scheme used to authenticate clients is described here:

http://ceph.newdream.net/wiki/Cephx

== Quality assurance ==

CEPH is a complex system, requiring multiple nodes and tuning to be utilized. The configuration is well documented, and the system is fairly straight forward to get running.

There are no major known bugs open against CEPH in Debian or Ubuntu.

The Debian maintainer keeps CEPH up to date with upstream, and is very responsive to inquiries. The packaging was originally done by upstream, and they have been active in keeping it up to date. The original author of CEPH, Sage Weil, is listed as an uploader.

The package does ship a test suite, and I've opened a bug, targetted at beta-1, to enable it:

https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/932895

Package has a correct debian/watch file.

== UI standards ==

N/A

== Dependencies ==

Build-Deps not in main:

libcrypto++ - Needs Investigation - Upstream code supports libnss, but upstream prefers crypto++ because of its C++ API superiority and some noted oddness in libNSS. Upstream is willing to start testing the NSS code more if that is the preferrable over putting libcrypto++ in main.

The NSS support was added for Fedora/RHEL in this issue:
http://tracker.newdream.net/issues/812

I opened a bug to enable the libnss support:

https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/932896

If this is done, libcrypto++ does not need to be in main.

google-perftools - These are highly useful and gaining popularity in high scale applications. Will prepare a separate MIR for them.

== Standards compliance ==

The packaging is relatively straightforward, being of the "pre debhelper 7" style, but still quite simple. There are no known FHS violations in the packages.

== Maintenance ==

CEPH is under active development by a dedicated division of DreamHost, which is actively engaged in the Ubuntu and OpenStack communities. The Debian maintainer is also quite active in addressing any concerns and keeping up with upstream. CEPH should not be an inordinate burden on Ubuntu as a whole to maintain.

Tags: canonistack
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ceph (Ubuntu):
status: New → Confirmed
James Troup (elmo)
tags: added: canonistack
Revision history for this message
Sage Weil (sage-newdream) wrote :

Just a note that

> The CEPH filesystem has had a detailed paper written describing its security model:
>
> http://www.ssrc.ucsc.edu/Papers/leung-storagess06.pdf

is incorrect. Andrew implemented a prototype but i twas never merged into the main code base, and it is unclear whether we would go with a similar design or something simpler.

Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Thanks Sage, I've updated the MIR text to reflect that.

description: updated
Changed in ceph (Ubuntu):
assignee: nobody → Jamie Strandboge (jdstrand)
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

In the interest of time, here are some preliminary items:

Pulls in libcrypto++ from universe (LP: #932896).

Also pulls in libfcgi, google-perftools, libunwind from universe during the build.

Uses python-support (in universe), should be converted to dh_python2.

Not lintian clean:
ceph_0.41-1ubuntu1_amd64.deb:
W: ceph: init.d-script-missing-start etc/init.d/ceph 4
N: 2 tags overridden (2 errors)

gceph_0.41-1ubuntu1_amd64.deb:
W: gceph: binary-without-manpage usr/bin/gceph

librbd1_0.41-1ubuntu1_amd64.deb:
W: librbd1: binary-without-manpage usr/bin/ceph-rbdnamer

radosgw_0.41-1ubuntu1_amd64.deb:
W: radosgw: init.d-script-missing-start etc/init.d/radosgw 4

Revision history for this message
Sage Weil (sage-newdream) wrote :

> > Pulls in libcrypto++ from universe (LP: #932896).

It can build against libnss instead, just need to pass --with-nss to configure and adjust debian/control. I'd prefer libcrypto++ because the nss APIs kind of suck, but it's security's call. We can use nss for all of our qa if you go that way.

> > Also pulls in libfcgi

Needed for radosgw, but not for RBD. You can build without radosgw with --without-radosgw.

> > google-perftools

Can build without this with --without-tcmalloc, but it is much faster with it, so ideally that could be pulled into main too.

IIRC there is a minimal package that has the tcmalloc libraries only... libtcmalloc-minimal0.

> > libunwind from universe

No idea where that is coming from....

> > Uses python-support (in universe), should be converted to dh_python2.

Looking into this.

I'm also cleaning up the lintian errors. Patches will hit the upstream ceph.git shortly.

Revision history for this message
Sage Weil (sage-newdream) wrote :

The lintian errors are cleaned up in ceph.git.. see commits 292898a8f79a916ba3e44171bcfe0993c4e32339 838a7618a7e804517e7570213348ba59e305d570 9678c0971b3a2050589c1960a3a01017466708aa for those that affect 0.41.

Revision history for this message
Sage Weil (sage-newdream) wrote :

The dh_python2 switchover is upstream in commit 7fdf25bc58212a9ebf65eaab05195c41705062ba.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :
Download full text (4.2 KiB)

Security review:

This is a pretty big code base with 340k+ lines of code and this review should only be considered a very shallow audit.

From my notes in the UDS session where members of the security team, the kernel team, the server team and ceph upstream were present:
From discussions with upstream at UDS:
 * librbd provides kvm remote block
 * librados provides S3 to replace swift
 * radosgw provides RESTful interface for S3 replacement
 * data is unencrypted on wire
 * kernel module is upstream as of 2.6.34
 * fuse module doesn't work on 32 bit
 * kernel client auth: secret in kernel key ring, then it authorizes per session
 * At UDS menioned QA started 6 months ago, meaning that it now has had almost a year
 * has testsuites. It would be nice to have them enabled in the builds (LP: #932895)
 * cluster expected to be run within the data center (private network usage) because there is no encryption
 * ipv6 is supported (should be able to use ippsec if desired, but not tested)
 * in terms of access controls: client authenticates to object/pool, clients only given access to that object/pool. Someone once suggested 'maat', but it was never merged.

In terms of code quality, spot-checking various places, I noticed that defensive coding is not done consistently throughout the codebase. For example, various parts of the codebase have a lot of string operations that don't use their more secure counterparts (sprintf() instead of snprintf(), strcpy instead of strncpy()). There are also a lot of low level memory operations some of which are not checking return codes. Eg, I found things like:
   *buf = (char*)malloc(l);
    strcpy(*buf, str.c_str());
I did not have time to check if these various places could be controlled by an attacker, but considering the amount of code and that defensive coding is not done consistently throughout the codebase, I think we can expect a non-trivial maintenance cost associated with supporting this for 5 years. Building using --without-radosgw should help with this maintenance cost.

Has the following compiler warning:
common/admin_socket_client.cc:166:19: warning: 'socket_fd' may be used uninitialized in this function [-Wuninitialized]
osd/PG.cc:1331:20: warning: variable 'plu' set but not used [-Wunused-but-set-variable]

Ships initscripts: /etc/init.d/ceph, /etc/init.d/radosgw and various daemons (ceph-mon, ceph-mds, ceph-osd) listen on the network. radosgw uses fastcgi is is meant to be used in combination with a webserver and does not run as root.

No dbus services, no setuid programs, no sudo fragments, no cron jobs

As mentioned, has a fuse module. Fuse is hard to get right and it isn't strategic. Let's not support it.

Has the option of using secure authentication between the nodes (cephx)

Upstream wiki states "Filesystems: Ceph performs best when using BTRFS, but other filesystems such as ext3 and ext4 can also be used." The kernel team had some concerns on btrfs at UDS surrounding the fact that the on disk format was not finalized and the recovery tool was non-existent. I'm told by the kernel engineer on my team that the disk format is 'pretty much set' but that the recovery tool is not robust and there c...

Read more...

Changed in ceph (Ubuntu):
assignee: Jamie Strandboge (jdstrand) → Clint Byrum (clint-fewbar)
status: Confirmed → In Progress
Revision history for this message
Sage Weil (sage-newdream) wrote :

Done:
- dh_python2 switch is done
- lintian warnings are fixed (except gceph one, which we aren't including)
- Man page updates are upstream, commits 4a4b7994e60f6796b5760b551b12ebc2e2c81b09 and 119dd5ae3d4523585d3f2a59eb436b1f0ad5b2cd.
- nss is fine; we'll switch to that for our internal qa.

Not done for v0.41:
- The compiler warnings are all fixed in v0.45.
- Yehuda is looking and (non-radosgw) malloc callsites and fixing those up. Those cleanups will go into v0.46.

Also:
- We'd prefer demoting ceph-fuse to not building it at all.
- Is it possible to build with radosgw but demote it to universe (like ceph-fuse)?

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Re radosgw> yes, but radosgw is what is pulling in libfcgi, which is not in main and if ceph is in main all its build depends must be in main.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

The changes in 0.41-1ubuntu2 implement the security team's recommendations sufficiently. Thanks!

Revision history for this message
Jamie Strandboge (jdstrand) wrote :
Download full text (3.1 KiB)

0.41-1ubuntu2 was accepted.

$ change-override.py -c main -s precise -t ceph
2012-04-12 20:55:31 INFO Creating lockfile: /var/lock/launchpad-change-override.lock
2012-04-12 20:55:40 INFO Override Component to: 'main'
2012-04-12 20:55:40 INFO 'ceph - 0.41-1ubuntu2/universe/admin' source overridden
Confirm this transaction? [yes, no] yes
2012-04-12 20:55:46 INFO Transaction committed.
2012-04-12 20:55:46 INFO Done.

$ change-override.py -c main -s precise librados-dev librados2 librbd-dev librbd1
2012-04-12 20:55:59 INFO Creating lockfile: /var/lock/launchpad-change-override.lock
2012-04-12 20:56:08 INFO Override Component to: 'main'
2012-04-12 20:56:08 INFO 'librados-dev-0.41-1ubuntu1/universe/libdevel/OPTIONAL' binary overridden in precise/amd64
2012-04-12 20:56:08 INFO 'librados-dev-0.41-1ubuntu1/universe/libdevel/OPTIONAL' binary overridden in precise/armel
2012-04-12 20:56:08 INFO 'librados-dev-0.41-1ubuntu1/universe/libdevel/OPTIONAL' binary overridden in precise/armhf
2012-04-12 20:56:08 INFO 'librados-dev-0.41-1ubuntu1/universe/libdevel/OPTIONAL' binary overridden in precise/i386
2012-04-12 20:56:08 INFO 'librados-dev-0.41-1ubuntu1/universe/libdevel/OPTIONAL' binary overridden in precise/powerpc
2012-04-12 20:56:09 INFO 'librados2-0.41-1ubuntu1/universe/libs/OPTIONAL' binary overridden in precise/amd64
2012-04-12 20:56:09 INFO 'librados2-0.41-1ubuntu1/universe/libs/OPTIONAL' binary overridden in precise/armel
2012-04-12 20:56:09 INFO 'librados2-0.41-1ubuntu1/universe/libs/OPTIONAL' binary overridden in precise/armhf
2012-04-12 20:56:09 INFO 'librados2-0.41-1ubuntu1/universe/libs/OPTIONAL' binary overridden in precise/i386
2012-04-12 20:56:09 INFO 'librados2-0.41-1ubuntu1/universe/libs/OPTIONAL' binary overridden in precise/powerpc
2012-04-12 20:56:09 INFO 'librbd-dev-0.41-1ubuntu1/universe/libdevel/OPTIONAL' binary overridden in precise/amd64
2012-04-12 20:56:09 INFO 'librbd-dev-0.41-1ubuntu1/universe/libdevel/OPTIONAL' binary overridden in precise/armel
2012-04-12 20:56:09 INFO 'librbd-dev-0.41-1ubuntu1/universe/libdevel/OPTIONAL' binary overridden in precise/armhf
2012-04-12 20:56:09 INFO 'librbd-dev-0.41-1ubuntu1/universe/libdevel/OPTIONAL' binary overridden in precise/i386
2012-04-12 20:56:09 INFO 'librbd-dev-0.41-1ubuntu1/universe/libdevel/OPTIONAL' binary overridden in precise/powerpc
2012-04-12 20:56:10 INFO 'librbd1-0.41-1ubuntu1/universe/libs/OPTIONAL' binary overridden in precise/amd64
2012-04-12 20:56:10 INFO 'librbd1-0.41-1ubuntu1/universe/libs/OPTIONAL' binary overridden in precise/armel
2012-04-12 20:56:10 INFO 'librbd1-0.41-1ubuntu1/universe/libs/OPTIONAL' binary overridden in precise/armhf
2012-04-12 20:56:10 INFO 'librbd1-0.41-1ubuntu1/universe/libs/OPTIONAL' binary overridden in precise/i386
2012-04-12 20:56:10 INFO 'librbd1-0.41-1ubuntu1/universe/libs/OPTIONAL' binary overridden in precise/powerpc
Confirm this transaction? [yes, no] yes
2012-04-12 20:56:15 INFO Transaction committed.
2012-04-12 20:56:15 INFO Done.

(don't worry about the versions with the binaries I promoted, it should get sorted...

Read more...

Changed in ceph (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.