kolla-ansible

Consolidate & Optimize Logging and Metrics Collection

Registered by Boris Lukashev

The Kolla-Ansible deployment breaks acquisition of log and metrics data into a large number of containers with a significant cumulative resource footprint and requiring separate maintenance loads for things like updating fluentd configs, host/container/database metrics collector configurations, etc.
Both of these data types can be collected, processed (via VRL), shipped, and committed to persistent storage using a single Rust binary configured to measure and collect logs from each of the services currently handled by various containers and paradigms. Vector (https://vector.dev/) is highly extensible, safe, and can be used to consolidate all LMA functions prior to the data store into a single workflow and container with a much smaller resource requirement than the script interpreters currently handling these workloads.

Blueprint information

Status:
Not started
Approver:
None
Priority:
Undefined
Drafter:
Boris Lukashev
Direction:
Needs approval
Assignee:
None
Definition:
New
Series goal:
None
Implementation:
Unknown
Milestone target:
None

Related branches

Sprints

Whiteboard

Host level collection:
```
Log0machine0 -> vector0machine0(transform to match current schema or update that altogether, dedup) -> Log store
Log1machine0 -> vector0machine0(transform to match current schema or update that altogether, dedup) -> Log store
Metric0machine0 -> vector0machine0(transform & buffer+sort) -> Metric store
Metric1machine0 -> vector0machine0(transform & buffer+sort) -> Metric store
...
Log0machine1 -> vector0machine1(transform to match current schema or update that altogether, dedup) -> Log store
Log1machine1 -> vector0machine1(transform to match current schema or update that altogether, dedup) -> Log store
Metric0machine1 -> vector0machine1(transform & buffer+sort) -> Metric store
Metric1machine1 -> vector0machine1(transform & buffer+sort) -> Metric store
```
Aggregate collection:
```
Logs[0..L]machine[0..M] -> vector0machine[0..M] -> vector0machineX(transform & dedup) -> Log store
Metric[0..L]machine[0..M] -> vector0machine[0..M] -> vector0machineX(transform & buffer+sort) -> Metric store
```

(?)

Work Items

This blueprint contains Public information 
Everyone can see this information.
Edit subscription

Subscribers

No subscribers.