Details About Feedback Reports

The Environment Report

Each feedback report starts with a short environment report. This tells us which application is reporting, what kind of instrumentation was used, and whether the program completed successfully, unsuccessfully, or crashed outright. Here is an example environment report, from which we know:

The reporting application is gaim-0.68-1.sam.1. HTTP_SAMPLER_APPLICATION_NAME gaim
Version 0.1 of the scalar-pairs instrumentation scheme was used, and ran at 1/100 sampling density. HTTP_SAMPLER_INSTRUMENTATION_TYPE scalar-pairs
The program completed successfully: exit status zero and no fatal signal. HTTP_SAMPLER_EXIT_STATUS 0
The report was sent using version 0.4.2 of the sampler feedback tools at the given time and date (GMT). HTTP_SAMPLER_VERSION 0.4.2
DATE 2003-09-13 00:52:21

The Samples Report

Next comes the samples report. If you’ve read about instrumentation schemes, you know that the measurements are collected into a set of predicate counters, one per each interesting thing that may or may not happen. The samples report identifies the loaded compilation units (*.o) and dumps out the predicate counters for all of the instrumentation sites in each unit.

Here is an example samples report (347KB uncompressed, but much smaller when you send it to us). A line that looks like an XML "<samples>" tag starts the instrumentation report for one compilation unit. The string of hexadecimal digits is the MD5 checksum of the source code for that unit. Lines that follow are a dump of counters for each instrumentation site in that unit. This report used scalar-pairs instrumentation, which tests a triple of predicates at each instrumentation site. So the counter dumps show a triple of values per site. Note that almost everything is zero. This is because we are sampling very sparsely. At a sampling rate of 1/100, there is a 99% chance that something that really happened exactly once will show up as 0 instead of 1.

It’s hard to make much sense of a raw samples report. Here at Bug Isolation Headquarters we resolve those reports by connecting each site back to the actual source code features being measured. You can do this yourself using the resolve-samples tool from the sampler-devel package, along with the *-samplerinfo package for the appropriate application. Here is an example resolved samples report (8.4MB, but never actually sent from you to us).

Each line corresponds to one instrumentation site. Taking line 12169, for example, we learn:

The MD5 code signature helps us find other reports from the same compilation unit. 387e443448bea998551e1bf9f10431c4
This site is inside Gaim’s Yahoo! plugin, on line 217 of source file yahoo.c in function yahoo_packet_read(). /usr/lib/sampler/sites/usr/lib/gaim/
Instrumentation compared variables pos and x pos
… and observed pos < x never, pos == x once, and pos > x fourteen times. 0

Taken by itself, this may not seem terribly informative. But when we start comparing these behaviors across many runs, and correlating those behaviors with program outcome, we can start to find patterns of program misbehavior that lead to bugs. Read more about statistical debugging to see how we find the few bug-inducing anomalies among all this data.

The Crash Report

If the program completed normally, then the environment and samples reports are all the feedback we receive. If the program ended by crashing, then an additional crash report provides more details about the failure.

The crash report starts with the list of shared libraries that were loaded at the time of failure. This can help us find bugs relating to bad versions of libraries or unusual plugins. Here is an example shared libraries report taken from a Nautilus crash.

The remainder of the crash report gives stack traces for all application threads at the time of failure. This is the standard information one would expect to see in, for example, a Bug Buddy crash report. Here are example stack traces for the main thread and for all threads, both taken from the same Nautilus crash.

Our application packages do not contain debug information. Thus, the stack traces contain function names but not the values of function arguments or local variables. This is intended to help protect your privacy. If you want to provide extra detail in your crash reports, feel free to install the corresponding *-debuginfo-* packages. However, you should only do this for applications that do not manipulate sensitive data (such as passwords).