Each feedback report starts with a short environment report. This tells us which application is reporting, what kind of instrumentation was used, and whether the program completed successfully, unsuccessfully, or crashed outright. Here is an example environment report, from which we know:
| The reporting application is gaim-0.68-1.sam.1. | HTTP_SAMPLER_APPLICATION_NAME | gaim |
| HTTP_SAMPLER_APPLICATION_VERSION | 0.68 | |
| HTTP_SAMPLER_APPLICATION_RELEASE | 1.sam.1 | |
| Version 0.1 of the scalar-pairs instrumentation scheme was used, and ran at 1/100 sampling density. | HTTP_SAMPLER_INSTRUMENTATION_TYPE | scalar-pairs |
| HTTP_SAMPLER_INSTRUMENTATION_VERSION | 0.1 | |
| HTTP_SAMPLER_SPARSITY | 100 | |
| The program completed successfully: exit status zero and no fatal signal. | HTTP_SAMPLER_EXIT_STATUS | 0 |
| HTTP_SAMPLER_EXIT_SIGNAL | 0 | |
| The report was sent using version 0.4.2 of the sampler feedback tools at the given time and date (GMT). | HTTP_SAMPLER_VERSION | 0.4.2 |
| DATE | 2003-09-13 00:52:21 |
Next comes the samples report. If you’ve read about instrumentation schemes, you know that the measurements are collected into a set of predicate counters, one per each interesting thing that may or may not happen. The samples report identifies the loaded compilation units (*.o) and dumps out the predicate counters for all of the instrumentation sites in each unit.
Here is an example samples report
(347KB uncompressed, but much smaller
when you send it to us). A line that looks like an XML
"<samples>" tag starts the instrumentation report for one
compilation unit. The string of hexadecimal digits is the MD5
checksum of the source code for that unit. Lines that follow are
a dump of counters for each instrumentation site in that unit.
This report used scalar-pairs instrumentation, which tests a
triple of predicates at each instrumentation site. So the counter
dumps show a triple of values per site. Note that almost
everything is zero. This is because we are sampling very
sparsely. At a sampling rate of 1/100, there is a 99% chance that
something that really happened exactly once will show up as
0
instead of 1
.
It’s hard to make much sense of a raw samples report. Here at Bug Isolation Headquarters we resolve those reports by connecting each site back to the actual source code features being measured. You can do this yourself using the resolve-samples tool from the sampler-devel package, along with the *-samplerinfo package for the appropriate application. Here is an example resolved samples report (8.4MB, but never actually sent from you to us).
Each line corresponds to one instrumentation site. Taking line 12169, for example, we learn:
| The MD5 code signature helps us find other reports from the same compilation unit. | 387e443448bea998551e1bf9f10431c4 |
| This site is inside Gaim’s Yahoo! plugin, on line 217 of source file yahoo.c in function yahoo_packet_read(). | /usr/lib/sampler/sites/usr/lib/gaim/libyahoo.so.sites |
| yahoo.c | |
| 217 | |
| yahoo_packet_read | |
Instrumentation compared variables pos and x… |
pos |
| x | |
… and observed pos < x never, pos == x once, and pos > x fourteen times. |
0 |
| 1 | |
| 14 |
Taken by itself, this may not seem terribly informative. But when we start comparing these behaviors across many runs, and correlating those behaviors with program outcome, we can start to find patterns of program misbehavior that lead to bugs. Read more about statistical debugging to see how we find the few bug-inducing anomalies among all this data.
If the program completed normally, then the environment and samples reports are all the feedback we receive. If the program ended by crashing, then an additional crash report provides more details about the failure.
The crash report starts with the list of shared libraries that were loaded at the time of failure. This can help us find bugs relating to bad versions of libraries or unusual plugins. Here is an example shared libraries report taken from a Nautilus crash.
The remainder of the crash report gives stack traces for all application threads at the time of failure. This is the standard information one would expect to see in, for example, a Bug Buddy crash report. Here are example stack traces for the main thread and for all threads, both taken from the same Nautilus crash.
Our application packages do not contain debug information.
Thus, the stack traces contain function names but not the values
of function arguments or local variables. This is intended to
help protect your privacy. If you want to provide extra detail in
your crash reports, feel free to install the corresponding *-debuginfo-*
packages. However, you should only
do this for applications that do not manipulate sensitive data
(such as passwords).