7.3 Fuzz Tools AFL Exercise

1. Introduction

The goal of this exercise is to give you experience with a modern fuzz testing tool, American Fuzzy Lop (AFL). Your goal is to cause failures in the form of crashes or hangs. These failures typically indicate that you have owned bits in the program that you were not intended to own.

More specifically, your goals are:

Learn about the fuzz testing tool AFL and how it works.
Use AFL to try to cause failures in two example programs.
Find the cause of the failures (i.e., debug the program being tested).

You will use AFL to test two programs written in C.

The basic steps are to

Compile the test program using AFL's compiler wrapper that adds instrumentation.
Run AFL on the test program.
When AFL finds a crash, it will save the input that caused it.
Debug the program to determine the cause(s) of the crash.

2. Exercise Instructions

This exercise will be done entirely on the command line. Launch the virtual machine, open terminal, and navigate to the exercises directory using cd ~/Desktop/EXERCISES/fuzzing_exercises. If you have a UNIX-based computer (Linux, MacOS, FreeBSD), you might be able to run this assignment locally on your computer, but you are responsible for making any changes that might be needed to make things work on your computer.

2.1 Report Program Exercise

The first test program is named report and is located in ~/Desktop/EXERCISES/fuzzing_exercises/log_exercise/report/; it supports queries on a database of flight log entries. You run report by typing a command of the form:

./report logfile1 logfile2 ...

The input log files have one entry per line. Two sample log files are provided, named log1 and log2. report starts by processing the log files and then reads query commands from standard input. You can see a summary of the commands for report by typing

> help

2.1.1 Try out the Program

To start, you should run make in the report directory to compile the program.

cd ~/Desktop/EXERCISES/fuzzing_exercises/log_exercise/report/
make

You can start the program using one or both of the sample log files, such as:

report log1 log2

Some sample commands to try are:

> select dte sel act hdd f/t lng
> range dte start 1/1/1989
> list

2.1.2 Compile the Report Program Using afl-gcc

In order to fuzz the program using AFL, we need to compile the program using AFL's wrapper around gcc (afl-gcc). Compiling with afl-gcc inserts the instrumentation into the binary that AFL depends on by default (AFL does have an option to run uninstrumented binaries in QEMU, but we won't be working with that today). Without instrumentation, AFL isn't able to determine the path(s) that were executed when an input is run.

If you open the Makefile, you will see that the syntax between afl-gcc and gcc are identical, allowing them to be interchanged.

To compile it with afl-gcc, just run:

make clean
make afl

2.1.3 Fuzzing stdin

There are two ways to run report with given input. The first way we are going to test is the standard input interface the program provides, and the second, which we will test next, is the parsing of the files containing the flight logs. To run AFL, we are going to use the following command from the ~/Desktop/EXERCISES/fuzzing_exercises/log_exercise/report directory:

afl-fuzz -i afl_stdin_seeds/ -o afl_stdin_out/ ./report_afl log1

-i specifies the directory containing our input seeds. If you open the files in the ./afl_stdin_seeds/ directory (cat ./afl_stdin_seeds/*), you'll see that they contain examples of the commands we entered into the report program's prompt.
-o specifies the directory where AFL will write the results of the fuzzing. That directory is currently empty.
./report_afl log1 specifies the executable we compiled using afl-gcc that has the instrumentation, and a log file to use. The report program requires a log file, but otherwise it isn't important for testing the stdin.

When we don't specify a file for AFL to use as input to the program (as we will do while fuzzing the log file interface), it defaults to sending the mutated input to the stdin of the target program. In this case, that input will be provided to the command prompt of the report program.

AFL will keep running until it is stopped; let it run for at least one cycle (~15 minutes) before stopping it.

Notes:

AFL may complain that your terminal window is too small; either make the window bigger, or just zoom out until the error message disappears.
If it says that "your system is configured to send core dump notifications..." when running afl-fuzz, then run the following commands:
```
sudo su root
[sudo] password for user: softwaresecurity
echo core >/proc/sys/kernel/core_pattern
exit
```
Then rerun the afl-fuzz command.

2.1.4 Analyzing the Results

After stopping AFL, we can take a look at the issues it found in the ./afl_stdin_out/ directory where AFL will have written the results of the testing. Within the ./afl_stdin_out/crashes directory are the "unique" inputs that caused the program to crash. AFL uses the instrumentation that afl-gcc embedded to try to only report unique crashes, however, it will often report at least some duplicate crashes.

In order to determine where the issues in the code are, it may be helpful to compile the program with the -fsanitize=address flag. You can use the debug target to do so:

cd ~/Desktop/EXERCISES/fuzzing_exercises/log_exercise/report 
clean
make debug

Then run the inputs in the ./afl_stdin_out/crashes/ directory using the following syntax:

cat ./afl_stdin_out/crashes/[input] | ./report_afl log1

Where [input] is one of the files in the crashes directory.

Notes:

If AFL reports many crashes, you may want to use the utility explained at the end of the instructions to help speed up the process of going through the inputs.
If you encounter a permissions error for the crashes directory, run chmod -R 777 [dir] on the directory.

2.1.5 Fuzzing the Log File Interface

You should have found at least two issues with how input from stdin is handled, now we are going to test how well it handles input from the log files. To do this, we will specify a file for AFL to write mutated input to, and then give the file to the report program to process:

cd ~/Desktop/EXERCISES/fuzzing_exercises/log_exercise/report 
make clean
make afl
afl-fuzz -i afl_logs_seeds/ -o afl_logs_out/ -f log.fuzz ./report_afl ./log.fuzz

-f specifies the file that AFL will write the mutated input to.
./report_afl ./log.fuzz specifies our target program, and the log file to process.

The seeds we have provided to AFL in the afl_logs_seeds/ directory are truncated versions of the log1 and log2 files that only contain the first ~20 lines. As before, try to let AFL run for at least one cycle (~8 minutes) before stopping it.

2.1.6 Analyzing the Results

The output directory ./afl_logs_out/ has the same structure as the one for fuzzing stdin did. Look at the crashes AFL found in the ./afl_logs_out/crashes/ directory and try to determine what the issues were. This time, because we were testing how it handles log files, we can run:

./report afl_logs_out/crashes/[input]

Where [input] is the crashing input we want to run. Again, you may find recompiling with make debug, as before, to be helpful.

2.2 JSON Parser Program

The second test program is a small JSON parsing library written in C. There is a simply frontend for it called jsonTester located in ~/Desktop/EXERCISES/fuzzing_exercises/json_exercise. It is run by typing a command in the form of:

jsonTester file.json

It will simply print out a a description of it's internal representation of the JSON data that was parsed.

2.2.1 Compile the JSON Parser

The second program we are going to fuzz is a JSON parser. Navigate to the ~/Desktop/EXERCISES/fuzzing_exercises/json_exercise directory. Compile the program using make afl.

cd ~/Desktop/EXERCISES/fuzzing_exercises/json_exercise
make afl

2.2.2 Fuzzing the JSON Parser

Using very similar syntax to what we used for the log file interface above, run:

afl-fuzz -i seeds -o output -f fuzz.json ./jsonTester fuzz.json

Here, we are passing the fuzzed input (fuzz.json) to the program. If you look in the ./seeds/ directory, there are several short JSON files that AFL uses as a starting point for fuzzing.

2.2.3 Analyzing the Results

You should have found at least two unique crashes. There will be fewer inputs AFL found to crash the program in ./output/crashes/, but it will still be helpful to recompile the program in debug mode:

make
cleanmake debug

3. Utility for Analyzing Results

Note: Before using this tool, you need to install the libtmux python package by running pip3 install libtmux.

You may want to use the ~/Desktop/EXERCISES/fuzzing_exercises/analyze.py utility to help with going through the results AFL found if there are a lot of them. Make sure to first compile the program you fuzzed with the -fsanitize=address flag as laid out above, otherwise this will not be too helpful.

The general syntax for using it is: analyze.py [debug_binary] [syntax] [crash_directory], where the [syntax] arguments specifies how the binary should be run. The example below shows how to use it to debug the crashes found by fuzzing the stdin of the report program.

cd ~/Desktop/EXERCISES/fuzzing_exercises/analyze.py
./analyze.py log_exercise/report/report_afl "cat {f} | {b} ./log_exercise/report/log1" log_exercise/report/afl_stdin_out/crashes/

# From a new terminal, as the output from the utility will say
tmux attach -t afl-debug

The {f} and {b} syntax represents the input File and the target Binary. Keep your original terminal window open, and press enter to cycle through all of the inputs. As you go through, the result of running the input will be shown on the left pane of the new terminal, and the input itself on the right pane.

If you run this tool multiple times, make sure you kill the tmux session between each:

tmux kill-session -t afl-debug