CDMI Support to Swift

Registered by Krishna Sankar

The SNIA CDMI is an object store interface[http://www.snia.org/tech_activities/standards/curr_standards/cdmi]. An RFC for CDMI media types is in last call status [http://www.ietf.org/id/draft-cdmi-mediatypes-02.txt]. Would like to explore if we can implement the CDMI interfaces to the swift object store.
[update 1/21/11] RFC [http://www.rfc-editor.org/internet-drafts/draft-cdmi-mediatypes-07.txt] approved & cdmi-capability / cdmi-container / cdmi-domain / cdmi-object / cdmi-queue are valid media types [http://www.iana.org/assignments/media-types/application/]

Blueprint information

Status:
Complete
Approver:
None
Priority:
Undefined
Drafter:
Krishna Sankar
Direction:
Needs approval
Assignee:
None
Definition:
Obsolete
Series goal:
None
Implementation:
Unknown
Milestone target:
None
Completed by
John Dickinson

Related branches

Sprints

Whiteboard

Goal : The goal is to make a feature-rich extensible data cloud control plane layer for SPs as well as for enterprises to build private data clouds; and because of the same interfaces, the enterprises can span and cloudburst as and when needed.

+ At which level should we implement the server: independent server, middle-ware, or within current servers (hybrid architecture)?
+ Good idea to develop and leverage a common middle-ware layer in order to support further APIs (WebDav?) (rf. b ii)

A few ideas:

a) Logistics Pragmatics
  i) Deliver basic framework with put, get & delete for Bexter ?
  ii) Add more features working with users/discussion at dev summit
  iii) Also compare the cloudfiles vs. CDMI/S3 and see what these add to cloudfiles and if the extra features are interesting
 [update 1/21/11] Read through CloudFiles API doc (http://www.rackspacecloud.com/cloud_hosting_products/files/api/)
  From a first glance, CDMI lacks CDN; CloudFiles lack object metadata, capability & of course, queue.
  So, IMHO, 70%-80% is a direct mapping, CloudFiles even has JSON !
  Will dig deeper ...

b) Frameworks
  i) REST framework & router -> Django or Pylon or Piston ?
  ii) Good idea to develop and leverage a common middleware layer, developed for the S3 API ?
  iii) AuthC/AuthZ can come from DevAuth or SwAuth frameworks. Would need list of groups, users and so forth when security is enabled.

c) Data Services
   i) Compression, dedup, encryption, CAS, content redundancy (infrastructure redundant replication ie multiple copies with no common infrastructure dependency),...
  ii) Metadata search of objects - for example for storing health records with a rich set of metadata and then the capability to search/retrieve based on the metadata. This will, of course, require different storage and interface mechanisms for metadata and the actual data

d) If the underlying mechanisms to support the data services are not available, we will add them to swift

(?)

Work Items

Dependency tree

* Blueprints in grey have been implemented.