Resource Management Daemon - Last Level Cache

This document describes the changes required to Nova Compute to support LLC allocation using RMD and RMD plugin for RDT CAT.

Intel RDT (Resource Director Technology) framework includes Cache Allocation Technology (CAT) and Memory Bandwidth Allocation (MBA) among others. These functions deliver visibility and control over how shared resources such as last-level cache and memory bandwidth are used by applications, virtual machines (VMs) and containers. LLC is CPU bound resource that can also be associated to a NUMA topology. On current generation systems there is one LLC per socket but we could have multiple separate LLC. With CAT, each LLC is measured in cache ways and a size of cache way is determined by total size of LLC in MB and number of cache ways available. The cache ways are logically partitioned using Class of Service (CLOS). Number of such available CLOS is platform dependent. Using the feature of CLOS, we can associate a segment of LLC for sole use by set of processes described using PIDs and set of cores using core IDs. The segment mapping also allows to dedicate LLC way in shared or overlapped manner between processes or cores.

The benefit of such a feature is the high priority process need not contend for cache ways with low priority processes. We can also isolate noisy neighbors to their own segment of LLC or oversubscribe cache ways on the platform to set of processes that require best-effort performance by providing cache ways in shared manner or with overlap to each other. LLC being a resource that is continuously sensitive to changing workloads requirements can now be fine-tuned during run-time of the workload. Currently the allocation of LLC ways to processes or cores with CLOS can be controlled using Linux resctrl file system starting kernel version 4.10 or with backward compatibility on kernel version less than 4.10 using intel-cat-cmt library that uses MSR registers.

We can derive two requirements for this feature that is fulfilled using RMD and RMD plugin for RDT CAT
1. QoS based static configuration of LLC ways to workloads described as set of process IDs or core IDs. LLC ways can be allocated in a dedicated or shared manner. LLC ways can also be reserved for system process like OS and infrastructure process like OVS, vRouter for NFV based deployments.
2. Policy based monitoring and fine-tuning of LLC ways assigned to workloads in a closed-loop manner.