Evolving System Stack for Persistent Memory: Device Characterization, Caching, and Sharing Perspectives

Abstract:

Data has been critical to human society, and data storage systems that hold the data of every human being are playing an important role in people's modern lives. These storage systems serve people's information in a variety of places, from mobile devices and laptops to large-scale data centers and the cloud. People rely on these storage systems for critical applications such as e-commerce, social media, health care, and artificial intelligence. In the modern world, people interact with data storage systems almost every moment. As a result, there is a high demand for data storage systems that are reliable, large, performant, and cost-effective.

To meet these demands, the system community has been working on high-performance, low-cost storage devices as well as software system stacks to manage them. For decades, numerous storage devices such as tape, DRAM, HDD, and SSD have been invented. Meanwhile, system stacks to manage these devices have evolved, ranging from primitive data structures (e.g., B-tree) to complex file systems and distributed storage systems. This dissertation focuses on Persistent Memory (PM), a new development in the storage device landscape. We investigate how to evolve the system stack for new PM devices.

In the first part of this thesis, we characterize a popular PM-based device -- the Intel Optane SSD. We investigate the performance of Optane SSDs in response to a variety of micro-experiments. We formalize an "unwritten contract", which includes rules that are essential for Optane SSD users to achieve optimal performance. We also use carefully designed experiments to reveal the internals of Optane SSDs. We reveal Optane SSD's internal parallelism, read-write scheduling mechanisms, and so on. These internals provide insights about the unwritten contract. Finally, we discuss implications of our device characterization.

In the second part, we investigate how classic caching performs on modern storage hierarchies, with new PM devices filling the gap between DRAM and SSD. We compare DRAM/PM/Low-latency SSDs/Flash SSDs quantitatively. Our analysis shows that modern hierarchies have a much smaller performance difference between neighboring layers than traditional hierarchies. We then examine caching on modern hierarchies. Caching manages data across two layers: a cache layer (faster but smaller) and a capacity layer (slower but larger). We find that classic caching, which directs as many accesses to the cache layer as possible (referred as the principle of maximizing hit rates), cannot effectively utilize the significant available performance in today's capacity layer (e.g., PM). Thus, we introduce three improvements (read around, admission rejection, and feedback-based offloading) that evolve classic caching for PM hierarchies.

In the final part of this thesis, we study how unique characteristics of PM influence the effectiveness of existing sharing mechanisms. We focus on a detailed setup of multi-tenant in-memory key-value caches. We first summarize the basic mechanisms (for example, resource usage accounting, interference analysis, etc.) used to achieve diverse sharing goals. We then investigate these mechanisms on PM. Our analysis shows that existing sharing mechanisms designed for DRAM/block devices do not readily translate to PM due to PM's unique characteristics such as 256B access granularity and severe and unfair interference between reads and writes. To address this issue, we present Nyx, a PM access regulation framework that is optimized for today's PM without special hardware support. Nyx introduces new software sharing mechanisms for PM that can be be used to easily and efficiently support sharing policies such as resource limiting, QoS, fair slowdown, and proportional sharing.

Full Paper: PDF   BibTeX

Publications