A tree- based storage of data on disk

Registered by Kostja Osipov

Use B-trees or fractal trees based disk store for historical data.

Use case: storing dialogues, chats, comets of social application.
In a social application, there are many scenarios when it's necessary to serve the latest
data with high speed, and keep a history of old data, to be accessible in rare cases.

Such cases include a web chat, a chat history of an instant messenger, Facebook timeline,
"what's new", "personal wall", "friends page", etc.

Data in these use cases is usually easy to shard, since sharding can be done on a per-user basis.
However, caching and predictable processing of this data can not be easily implemented by means of
a standard database caching algorithm, since two different QoS (quality of service) levels apply to the "hottest"
data (best kept in RAM and served immediately) and "historical" data, which can be served with a help of a few
disk seeks, if necessary.

To support this use case, it should be possible to define spaces which store their data, perhaps partially, on disk.

Add the notion of space types (in-memory, disk) and a new space type: "comet".
"Comet" space type presumes that some fields of a tuple are kept permanently in memory,
and others are pushed out to a disk-based data store.
A single timeline or comet is thus stored in a single tuple, indexed by user id.
It is sufficient that in-memory storage can be requested only for the first few fields of a tuple.
It is not necessary to be able to create indexes on fields which are disk-based.

For the disk-based tail, we should implement some standard tree-based caching scheme,
such as B-trees, LSM trees, fractal trees (to be decided).
The cache search/write process can be implemented using the same pattern
as the thread based WAL writer, except it can be completely async: we keep
writing to the WAL all changes to the tuples, and WAL continues to serve as the
foundation of D in ACID mechanism.

We will need to change tuple format, to support tuples which are partially stored on disk.

Blueprint information

Status:
Not started
Approver:
None
Priority:
Undefined
Drafter:
None
Direction:
Needs approval
Assignee:
None
Definition:
New
Series goal:
Accepted for 1.5
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.

Subscribers

No subscribers.