Optimizing Modern Heterogeneous Systems: From Storage to Memory Kaiwei Tu Department of Computer Sciences University of Wisconsin-Madison

Abstract:

Heterogeneous storage and memory systems combine multiple device types to balance performance, capacity, and cost, typically pairing a faster, smaller performance device with a slower, larger capacity device. For decades, these systems have been hierarchical: the performance device is orders of magnitude faster, so the capacity device's performance can be safely ignored. Modern devices break this assumption. The emergence of low-latency SSDs, persistent memory, and CXL memory has greatly narrowed the performance gap between neighboring devices, producing non-hierarchical storage and memory systems in which the capacity tier offers comparable bandwidth and, under contention, can even be faster than the performance tier. Both classic and state-of-the-art methods fall short on non-hierarchical architectures. For example, caching, which replicates hot data in the performance device, and tiering, which migrates hot data to it, both optimize for the performance tier; once that tier saturates, the capacity tier remains underutilized and overall performance is bounded by a single device. This dissertation addresses this challenge in three parts, each identifying the limitations of classic and state-of-the-art approaches in non-hierarchical storage and memory systems and proposing new designs that aim to dynamically utilize all devices.

The first part revisits caching for modern storage hierarchies. We introduce Non-Hierarchical Caching (NHC), an approach that layers a lightweight, feedback-driven scheduler on top of an existing cache controller. The key insight is that a cache hit need not be served from the performance device: when that device saturates, NHC redirects a fraction of hits to the copy on the capacity device, exploiting its unused bandwidth. We implement NHC as Orthus-CAS, a kernel module within the open-source Open CAS framework, with configurable optimization targets including throughput, average latency, and tail latency. Compared to classic caching, Orthus-CAS achieves up to 2x higher throughput when the performance device is saturated, and it substantially outperforms prior cache-bypass approaches under dynamic workloads.

The second part redesigns tiering, which avoids caching's duplicated capacity but keeps a single copy of each block and must migrate data to shift load. We introduce Mirror-Optimized Storage Tiering (MOST), a tier-based approach that mirrors a small amount of hot data across devices and balances load by routing requests among the copies instead of migrating data. A simple, self-adjusting optimization mechanism controls the routing ratio online, requiring no prior knowledge of the workload or devices. We implement MOST as Cerberus, a storage management layer within the widely used flash cache library CacheLib. Across two storage hierarchies, Cerberus achieves up to 2.34x higher throughput and reduces P99 latency by up to 75% compared to state-of-the-art systems on static workloads, improves throughput by up to 1.86x across four production cache workloads, and reduces device writes by up to 84% under dynamic workloads.

The third part extends load balancing to heterogeneous memory, where local DRAM is paired with second-level memory such as CXL devices and memory-interconnect contention can invert the latency relationship between tiers. Existing contention-aware systems rebalance by migrating pages, but migration adds traffic to the already contended memory subsystem, slows convergence, interferes with foreground requests, and repeats data movement under bursty load. We introduce Hecaton, a heterogeneous memory management system built on mapping-based load balancing: Hecaton maintains mirrored copies of read-mostly pages across tiers and balances load through page table remapping without data copying under contention. A lightweight latency-feedback controller, low-overhead reactive and proactive mirroring, and a write-recency test for selecting pages worth mirroring make this approach practical. Across microbenchmarks, bursty workloads, and realistic applications, Hecaton outperforms contention-oblivious tiering by up to 2.60x and the best migration-based baseline by 1.64x, converges up to 32.8x faster than the lowest-interference migration variant, and migrates up to 34x less data.

Together, these three systems introduce new designs that make better use of all devices in non-hierarchical storage and memory systems.

Full Paper: PDF   BibTeX

Publications