Running Benchmarks

Using PCM-disk

PCM-disk is an emulator for a PCM-based block device, which can be used to compare Mnemosyne against other uses of PCM. Based on Linux's RAM-disk (brd device driver), PCM-disk introduces delays when writing a block.

Loading

PCM-disk is implemented as a kernel module. To use PCM-disk, the block device nodes /dev/pcm0 and /dev/pcm0-ctl must exist. To create the necessary device nodes, you may use the provided $MNEMOSYNE/kernelmode/pcmdisk/dev.sh script.

To load the PCM-disk, create an ext2 file system on it and mount the file system on /mnt/pcmfs, you may use the provided $MNEMOSYNE/kernelmode/pcmdisk/fs.sh script:

% cd $MNEMOSYNE/kernelmode/pcmdisk
% ./fs.sh -c

To unmount the file system and unload PCM-disk, you may use the same script:

% cd $MNEMOSYNE/kernelmode/pcmdisk
% ./fs.sh -r

Configuring

To dynamically configure and control PCM-disk, you may use the provided pcmdiskctl user-mode application.

pcmdiskctl accepts the following command line options:

• --print-config: Prints the current PCM-disk configuration
• --set-pcm-bw=: Sets emulated PCM bandwidth (in MB/s)
• --set-dram-bw=: Sets DRAM system bandwidth (in MB/s)
• --reset-stat : Resets statistics
• --print-stat : Prints statistics

Calibrating

PCM-disk basically emulates PCM by introducing some latency after writing a block to handicap the available bandwidth to the DRAM system. The two parameters --set-pcm-bw and --set-dram-bw may be used to control the available bandwidth. You may follow the following procedure to configure and calibrate the available bandwidth:

1. Measure the maximum DRAM bandwidth of your system using the tool $MNEMOSYNE/usermode/tool/bandwidth-ram. This will be the value passed to --set-dram-bw .
2. Reset PCM-disk statistics via --reset-stat.
3. Set the target PCM bandwidth via --set-pcm-bw .
4. Run your benchmark.
5. Check the emulated PCM-disk bandwidth (PCM-BW) by getting a PCM-disk statistics report via --print-stat.
6. Calibrate by increasing/decreasing the target PCM bandwidth till the emulated PCM-disk bandwidth is close to the targeted value.

Todo:

PCM-disk should use a feedback loop to self-calibrate.

These are the bandwidth values for each latency used in the ASPLOS 2011 paper:

• 150ns: --set-dram-bw=7000 --set-pcm-bw=5000 --> emulated PCM-BW ~= 4000 MB/s
• 1000ns: --set-dram-bw=7000 --set-pcm-bw=800 --> emulated PCM-BW ~= 800 MB/s
• 2000ns: --set-dram-bw=7000 --set-pcm-bw=400 --> emulated PCM-BW ~= 400 MB/s

Mnemosyne Runtime Configuration

Mnemosyne can be dynamically configured through mnemosyne.ini, which must be placed in the working directory. Configuration parameters include:

libmcore library

• reset_segments : Clears persistent regions upon restart. Can be used to force a clean start of the application. Default is false.
• segments_dir: The directory where the files backing the persistent regions are placed. Default is $CWD/.segments.

libmtm library

• force_mode: Sets the transaction execution mode. Execution modes include pwbetl (durable w/ locking) and pwbnl (durable w/o locking). Default is pwbetl.
• stats : Enables statistics collection. Library must be compiled with statistics support. Default is false.

An example configuration file:

mcore: 
{
    segments_dir = "/tmp/segments";
}

mtm:
{
    force_mode = "pwbetl";
    stats = false;
}

Linux Page Cache Tuning

When running Mnemosyne experiments make sure the Linux kernel page cache is not asynchronously flushing dirty pages. This is not required with SCM and Mnemosyne as page data and mappings are persistent. Additionally, the unnecessary asynchronous flushing of dirty pages causes disk I/O, which disturbs the experiments.

As an extreme, you may set the following watermarks:

• echo 100 > /proc/sys/vm/dirty_background_ratio
• echo 100 > /proc/sys/vm/dirty_ratio
• echo 10000 > /proc/sys/vm/dirty_expire_centisecs

Running KVStore

We use kvdriver ($MNEMOSYNE/usermode/bench/kvstore/kvdriver) to exercise four key-value stores (persistent hashtable) implementations:

• kvdriver-ht : exercises Berkeley DB's hash table (bdb) running on PCM-disk and a C Hash Table (ht) by Christopher Clark extended to support multiple threads and Mnemosyne persistence.
• kvdriver-tc-base: exercises the base version of TokyoCabinet running on PCM-disk
• kvdriver-tc-mtm: exercises the Mnemosyne based version of TokyoCabinet

To measure the scalability of the persistence solutions, the workload generated by kvdriver is such that threads do not touch the same data to avoid true conflicts due to sharing. Threads may still experience false conflicts due to scalability bottlenecks of the persistence solution. For example, Mnemosyne may experience false conflicts on the transactional locks.

kvdriver accepts the following command line options:

• --ubench=: Microbenchmark to run (please see below)
• --runtime=: Runtime of the experiement in seconds
• --nthreads=: Number of threads
• --vsize=: Value size
• --nkeys=: Number of keys in the hashtable

For kvdriver-ht, some microbenchmarks of interest are:

• mtm_mix_latency_etl: measures the latency of inserting/deleting elements to a persistent hash table
• mtm_mix_throughput_etl: measures the throughput of inserting/deleting elements to a persistent hash table
• bdb_mix_latency: measures the latency of inserting/deleting elements to a BDB hash table
• bdb_mix_throughput: measures the throughput of inserting/deleting elements to a BDB hash table

For kvdriver-tc-base, some microbenchmarks of interest are:

• tc_base_mix_latency: measures the latency of inserting/deleting elements to the base TC hash table
• tc_base_mix_throughput: measures the throughput of inserting/deleting elements to the base TC hash table

For kvdriver-tc-mtm, some microbenchmarks of interest are:

• tc_mtm_mix_latency_etl: measures the latency of inserting/deleting elements to a Mnemosyne based TC hash table
• tc_mtm_mix_throughput_etl: measures the throughput of inserting/deleting elements to a Mnemosyne based TC hash table

Running OpenLDAP

Scripts

The accompanying scripts $MNEMOSYNE/usermode/bench/openldap/driver/scripts/mnemosyne may be used to config and run the Mnemosyne-based version of OpenLDAP. Please ensure that:

• the script variables OPENLDAP_BUILD and MNEMOSYNE_PATH point to the OpenLDAP build and Mnemosyne tree respectively.
• the variable include in $MNEMOSYNE/usermode/bench/openldap/driver/scripts/mnemosyne/slapd.conf points to the right schema file.
• the IP address and port number correspond to the one of your OpenLDAP server
• in the case you use slapd.conf for running other backends, ensure the variable directory points to a valid directory (preferably under a file system backed by PCM-disk).

The accompanying scripts $MNEMOSYNE/usermode/bench/openldap/driver/scripts/query may be used to test OpenLDAP with simple add/search queries.

SLAMD

We recommend using the SLAMD Distributed Load Generation Engine (http://www.slamd.com) to generate a workload for OpenLDAP. You may use the <it>Add/Remove Rate Template</it> configured using the parameters provided in $MNEMOSYNE/usermode/bench/openldap/driver/slamd/slamd-template.

---