SHORE Storage Manager: The Multi-Threaded Version

Release 6.0-Beta.4

Description

This is an experiment test-bed library for use by researchers who wish to write multi-threaded software that manages persistent data.

This storage engine provides the following capabilities:

• transactions with ACID properties, with ARIES-based logging and recovery, primitives for partial rollback, transaction chaining, and early lock release,
• prepared-transaction support for two-phased commit,
• persistent storage structures : B+ tree indexes, R* trees (spatial indexes), and files of untyped records,
• fine-grained locking for records and B+ tree indexes with deadlock detection, optional lock escalation and optional coarse-grained locking,
• in-memory buffer management with optional prefetching,
• extensible statistics-gathering, option-processing, and error-handling facilities.

This software runs on Pthreads, thereby providing its client software (e.g., a database server) multi-threading capabilities and resulting scalability from modern SMP and NUMA architectures, and has been used on Linux/x86-64 and Solaris/Niagara architectures.

Background

The SHORE (Scalable Heterogeneous Object REpository) project at the University of Wisconsin - Madison Department of Computer Sciences produced the first release of this storage manager as part of the full SHORE release in 1996. The storage manager portion of the SHORE project was used by other projects at the UW and elsewhere, and was intermittently maintained through 2008.

The SHORE Storage Manager was originally developed on single-cpu Unix-based systems, providing support for "value-added" cooperating peer servers, which were processes on single-CPU machines. One such server was the SHORE Value-Added Server (http://www.cs.wisc.edu/shore), and another of which was Paradise (http://www.cs.wisc.edu/paradise) at the University of Wisconsin. The TIMBER (http://www.eecs.umich.edu/db/timber, and http://dx.doi.org/10.1007/s00778-002-0081-x) and Pericope (http://www.eecs.umich.edu/periscope) projects at the University of Michigan, PREDATOR (http://www.distlab.dk/predator) at Cornell, Lachesis (http://www.vldb.org/conf/2003/papers/S21P03.pdf) BODHI (http://dsl.serciisc.ernet.in/projects/BODHI/homepage/index.html, http://doi.acm.org/10.1145/1007568.1007717) used the SHORE Storage Manager. The storage manager has been used for innumerable published studies since then.

The storage manager had its own (non-preemptive) threads library, which forked a per-disk cooperating process to provide non-blocking I/O. The storage manager also had a communications library for peer-server communication. Until recently, the code structure, nomenclature, and contents reflected its SHORE roots.

In 2007, the Data Intensive Applications and Systems Labaratory (DIAS) at Ecole Polytechnique Federale de Lausanne began work on a port of release 5.0.1 of the storage manager to Pthreads, and developed more scalable synchronization primitives, identified bottlenecks in the storage manager, and improved the scalability of the code. This work was on a Solaris/Niagara platform and was released as Shore-MT http://diaswww.epfl.ch/shore-mt). It was a partial port of the storage manager and did not include documentation. Projects using Shore-MT include StagedDB/CMP (http://www.cs.cmu.edu/~stageddb/), DORA (http://www.cs.cmu.edu/~ipandis/resources/pvldb10pandis.pdf)

In 2009, the University of Wisconsin - Madison (UW) took the first Shore-MT release and ported the remaining code to Pthreads. This work as done on a Red Hat Linux/x86-64 platform. The UW and DIAS cooperated on bug fixes and some internal rewrites. This release is the result of that work, and includes this documentation, bug fixes, and supporting test code.

Copyrights and Licenses

This distribution contains code and documentation subject to several copyrights and licenses. See the copyrights page Copyrights and Licenses, here.

Getting Started With the Shore Storage Manager

A good place to start is with the programming interface, described here.

Configuring and Building the Storage Manager

See this page to configure and build the storage manager.

Implementation Notes

See this page for some implementation details.

Reporting Bugs

This is unsupported software. We hope there will be times when someone will be able to respond to problem reports. A problem report should include the following information:

• Release number/version of the storage manager.
• Platform (operating system, hardware/CPU type).
• Configuration options (command used to build the storage manager).
• Copy of program that demonstrates problematic behavior, if possible (if not itself part of the release).
• Description of problematic behavior, and how the program is run to reproduce the behavior. If you can create a script to reproduce the problem, include that.

Mail this to: shore-mt@cs.wisc.edu

References

See this page for references to selected papers from which ideas are used in the Shore Storage Manager.

---