This page goes into a bit more technical depth about the Cooperative Bug Isolation Project, how it works, and what it means for you to participate. If you have no idea what the Cooperative Bug Isolation Project is, or why you are reading this page, please start at the beginning. If you find that you still have questions after reading this page, let us know so that we can make it better.
The software in our downloads area has been augmented with extra instrumentation: special code that runs along side the application to monitor its behavior. The specific behavior we monitor varies depending on how the application was built. In general, we are looking at data values and decisions within the application, and testing them to see if they show unusual patterns. If monitoring picks up something unusual, and the application also crashes, then we may have isolated the cause of a bug. More details about instrumentation schemes.
Recording everything an application does would be far too slow and could also be a serious invasion of privacy. Instead, we only look at a very sparse sample of what is going on. The instrumentation we add is not active all the time. It turns on and off randomly while the application runs, with most of its time spent off. That gives us a partial but fair picture of what the code is doing without slowing you down. You can participate in this project without worrying that our extra monitoring will get in your way. More details about sparse random sampling.
Before each instrumented application exits, it sends a feedback report up to our collection server here at Bug Isolation Headquarters. That report includes:
If the program received a fatal signal, then the feedback report also includes crash details:
Feedback reports do not include your name, the name of your computer, your credit card numbers, or any other personally identifying information. We take your privacy very seriously, and collect only the information we need to conduct our research and isolate bugs. More details about feedback report contents and interpretation.
Because the sampling is so sparse, we don’t learn very much about any single run. But over time, with enough people, we build up an aggregate picture of how the software behaves overall. We also build up an aggregate picture of how the software misbehaves.
When your program crashes, something went wrong. Something unusual happened that the developers never thought of. There is something different about the way the program behaved versus all those other runs that didn’t crash. That difference is what we’re looking for. Among all of the potentially interesting things we counted, we’re looking for the ones that change between runs that crash and runs that don’t crash. That’s why we need information about both good runs and bad ones. The good runs tell us what the software is supposed to be doing. The bad ones tell us what went wrong by revealing how they are different from the good runs.
This is what we call statistical debugging: finding bugs in programs via automated statistical analysis instead of laborious manual inspection. Our strategies combine facets of traditional program analysis with innovative techniques from statistics and machine learning theory. We cannot say too much more at this point, as this is the facet of our research that is changing most rapidly: anything we post here will be obsolete by the time you read it. The curious machine learning researcher is advised to take a look at our published work for a more detailed treatment of the specific statistical modeling techniques we use.
Your privacy is very important to us. We learn from overall patterns in the data, so this project can only succeed if a large number of users feel comfortable participating. We have taken several measures to protect your privacy and maintain the security of the data we collect.
No data is ever collected without your permission. Data is managed securely on your machine, encrypted for transit, and stored only on secure, firewall-protected servers here at Bug Isolation Headquarters for analysis. Even if any part of this system is compromised, there is very little information one can learn from any single report. This is a consequence of our sparse random sampling: there is simply too little information about any single run to be useful to an attacker. Our research attempts to find patterns in hundreds or thousands of runs. Reading a single run is like predicting the weather by catching a single raindrop. More details about privacy measures.
Several published papers are available which describe the project’s goals and implementation in greater detail.
If you would rather read what other people think of our work, see:
A Better Way to Beta: Prof’s program is helping software designers work the bugs out
This research began as part of the Open Source Quality Project in the Computer Science Division at UC Berkeley.