Analyzing Chopper Outputs

In this document, we describe how to reproduce the figures in paper Reducing File System Latencies with Chopper. More importantly, the document provides some basic code that you can modifiy to analyze new results from Chopper.

I try to make it easy for you to run the code. The minimum requirement is that you have R and several packages installed (describe in this page). Then you can simply copy and paste the code to R and you will see the figures. This may take some time as it has to download data files from GitHub. If you want it to be faster, you can also download Chopper at https://github.com/junhe/chopper and produce the figures locally (instructions are also available in this page).

Please note that some factor names in source code are different to the paper. And I am too lazy to keep them consistent. But you should map them fairely easy. I provide a mapping table here.

Paper	Code
DiskSize	disk.size
UsedRatio	disk.used
FreeSpaceLayout	layoutnumber
CPUCount	num.cores
FileSize	file.size
ChunkCount	num.chunk
InternalDensity	Fullness
ChunkOrder	chunk.order
Fsync	fsync
Sync	sync
FileCount	num.files
DirectorySpan	dir.span

Install R

We wrote the analysis with R. You can download and install R from http://www.r-project.org/ .

If you use Ubuntu, you can install it by:

sudo apt-get install r-base

Install Required Packages for Analysis

Run the following script in R to install necessary packages:

install.packages(c("grid", "ggplot2", "combinat", "gridExtra", "plyr", "reshape2", "R.utils", "devtools"), dependencies = TRUE)

If the one above does not work, try the following. I almost ruined my R environment when trying to verify the command. I am not gonna try it again… You get the idea - install these packages listed.

install.packages(c("grid", "ggplot2", "combinat", "gridExtra", "plyr", "reshape2", "R.utils", "devtools"))

Produce Figures

We now describe how to reproduce the figures in paper Reducing File System Tail Latencies with Chopper.

Figure 1

You can copy and paste the following code to R to reproduce the figure without downloading the analysis code.

library(devtools)
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/fig-1.layoutnumber.R')

If you have downloaded Chopper to your local file system, you can also copy and paste the following code to R to get the figure.

USE.REMOTE.CHOPPER = FALSE
setwd('/Users/junhe/workdir/chopper/vis-analysis') # Replace the path with yours
source('fig-1.layoutnumber.R')

Figure 2

Figure 2 is diagram showing components of Chopper. So not here.

Figure 3

Copy and paste the following code to R to reproduce figure 3 without local Chopper folder.

library(devtools)
USE.REMOTE.CHOPPER = TRUE
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/common.r')
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/fig-3.cdfs-ext4-xfs.r')

If you have downloaded Chopper to your local file system, you can also copy and paste the following code to R to get the figure.

USE.REMOTE.CHOPPER = FALSE
setwd('/Users/junhe/workdir/chopper/vis-analysis') # Replace the path with yours
source('./common.r')
source('./fig-3.cdfs-ext4-xfs.r')

Figure 4

Copy and paste the following code to R to reproduce the figure without local Chopper folder.

library(devtools)
USE.REMOTE.CHOPPER = TRUE
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/common.r')
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/fig-4.variance.r')

If you have downloaded Chopper to your local file system, you can also copy and paste the following code to R to get the figure.

USE.REMOTE.CHOPPER = FALSE
setwd('/Users/junhe/workdir/chopper/vis-analysis') # Replace the path with yours
source('./common.r')
source('./fig-4.variance.r')

Figure 5

Copy and paste the following code to R to reproduce the figure without local Chopper folder.

library(devtools)
USE.REMOTE.CHOPPER = TRUE
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/common.r')
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/fig-5.tail-distribution.r')

If you have downloaded Chopper to your local file system, you can also copy and paste the following code to R to get the figure.

USE.REMOTE.CHOPPER = FALSE
setwd('/Users/junhe/workdir/chopper/vis-analysis') # Replace the path with yours
source('./common.r')
source('./fig-5.tail-distribution.r')

Figure 6

Copy and paste the following code to R to reproduce the figure without local Chopper folder.

library(devtools)
USE.REMOTE.CHOPPER = TRUE
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/common.r')
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/fig-6.tail-interactions.r')

If you have downloaded Chopper to your local file system, you can also copy and paste the following code to R to get the figure.

USE.REMOTE.CHOPPER = FALSE
setwd('/Users/junhe/workdir/chopper/vis-analysis') # Replace the path with yours
source('./common.r')
source('./fig-6.tail-interactions.r')

Figure 7

Copy and paste the following code to R to reproduce the figure without local Chopper folder.

library(devtools)
USE.REMOTE.CHOPPER = TRUE
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/common.r')
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/fig-7.vanilla-vs-final.r')

If you have downloaded Chopper to your local file system, you can also copy and paste the following code to R to get the figure.

USE.REMOTE.CHOPPER = FALSE
setwd('/Users/junhe/workdir/chopper/vis-analysis') # Replace the path with yours
source('./common.r')
source('./fig-7.vanilla-vs-final.r')

Figure 8

Copy and paste the following code to R to reproduce the figure without local Chopper folder.

library(devtools)
USE.REMOTE.CHOPPER = TRUE
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/common.r')
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/fig-8.effects.r')

If you have downloaded Chopper to your local file system, you can also copy and paste the following code to R to get the figure.

USE.REMOTE.CHOPPER = FALSE
setwd('/Users/junhe/workdir/chopper/vis-analysis') # Replace the path with yours
source('./common.r')
source('./fig-8.effects.r')

Figure 9a

Copy and paste the following code to R to reproduce the figure without local Chopper folder.

library(devtools)
USE.REMOTE.CHOPPER = TRUE
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/common.r')
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/fig-9a.randomness-cdf.r')

If you have downloaded Chopper to your local file system, you can also copy and paste the following code to R to get the figure.

USE.REMOTE.CHOPPER = FALSE
setwd('/Users/junhe/workdir/chopper/vis-analysis') # Replace the path with yours
source('./common.r')
source('./fig-9a.randomness-cdf.r')

Figure 9bc

Copy and paste the following code to R to reproduce the figure without local Chopper folder.

library(devtools)
USE.REMOTE.CHOPPER = TRUE
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/common.r')
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/fig-9bc.randomness-interaction.r')

If you have downloaded Chopper to your local file system, you can also copy and paste the following code to R to get the figure.

USE.REMOTE.CHOPPER = FALSE
setwd('/Users/junhe/workdir/chopper/vis-analysis') # Replace the path with yours
source('./common.r')
source('./fig-9bc.randomness-interaction.r')

Figure 10a

Copy and paste the following code to R to reproduce the figure without local Chopper folder.

library(devtools)
USE.REMOTE.CHOPPER = TRUE
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/common.r')
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/fig-10a.effects-of-removing.r')

If you have downloaded Chopper to your local file system, you can also copy and paste the following code to R to get the figure.

USE.REMOTE.CHOPPER = FALSE
setwd('/Users/junhe/workdir/chopper/vis-analysis') # Replace the path with yours
source('./common.r')
source('./fig-10a.effects-of-removing.r')

Figure 10bc

Copy and paste the following code to R to reproduce the figure without local Chopper folder.

library(devtools)
USE.REMOTE.CHOPPER = TRUE
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/common.r')
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/fig-10bc.effects-of-removing.r')

If you have downloaded Chopper to your local file system, you can also copy and paste the following code to R to get the figure.

USE.REMOTE.CHOPPER = FALSE
setwd('/Users/junhe/workdir/chopper/vis-analysis') # Replace the path with yours
source('./common.r')
source('./fig-10bc.effects-of-removing.r')

Figure 11

Copy and paste the following code to R to reproduce the figure without local Chopper folder.

library(devtools)
USE.REMOTE.CHOPPER = TRUE
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/common.r')
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/fig-11.xSDxSE.r')

If you have downloaded Chopper to your local file system, you can also copy and paste the following code to R to get the figure.

USE.REMOTE.CHOPPER = FALSE
setwd('/Users/junhe/workdir/chopper/vis-analysis') # Replace the path with yours
source('./common.r')
source('./fig-11.xSDxSE.r')

Figure 12

Copy and paste the following code to R to reproduce the figure without local Chopper folder.

library(devtools)
USE.REMOTE.CHOPPER = TRUE
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/common.r')
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/fig-12.xSDxSExSG.r')

If you have downloaded Chopper to your local file system, you can also copy and paste the following code to R to get the figure.

USE.REMOTE.CHOPPER = FALSE
setwd('/Users/junhe/workdir/chopper/vis-analysis') # Replace the path with yours
source('./common.r')
source('./fig-12.xSDxSExSG.r')

Figure 13

Copy and paste the following code to R to reproduce the figure without local Chopper folder.

library(devtools)
USE.REMOTE.CHOPPER = TRUE
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/common.r')
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/fig-13.norm-bug.r')

If you have downloaded Chopper to your local file system, you can also copy and paste the following code to R to get the figure.

USE.REMOTE.CHOPPER = FALSE
setwd('/Users/junhe/workdir/chopper/vis-analysis') # Replace the path with yours
source('./common.r')
source('./fig-13.norm-bug.r')

Figure 14

Copy and paste the following code to R to reproduce the figure without local Chopper folder.

library(devtools)
USE.REMOTE.CHOPPER = TRUE
source_url('https://raw.githubusercontent.com/junhe/chopper/master/vis-analysis/fig-14.latency-reduction.r')

If you have downloaded Chopper to your local file system, you can also copy and paste the following code to R to get the figure.

USE.REMOTE.CHOPPER = FALSE
setwd('/Users/junhe/workdir/chopper/vis-analysis') # Replace the path with yours
source('./common.r')
source('./fig-14.latency-reduction.r')

Analyzing Chopper Outputs

Jun He

Install R

Install Required Packages for Analysis

Produce Figures

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9a

Figure 9bc

Figure 10a

Figure 10bc

Figure 11

Figure 12

Figure 13

Figure 14