An ensemble consists of a set of independently trained classifiers (such as neural networks or decision trees) whose predictions are combined when classifying novel instances. Previous research has shown that an ensemble as a whole is often more accurate than any of the single classifiers in the ensemble. Bagging and Boosting are two relatively new but popular methods for producing ensembles. In this talk I will present an empirical study of these methods on 23 data sets using both neural networks and decision trees as the classification algorithm. The results clearly indicate a number of conclusions. First, while Bagging is almost always more accurate than a single classifier, it is often less accurate than Boosting. On the other hand, Boosting sometimes creates ensembles that are less accurate than a single classifier. Analysis indicates that the performance of the Boosting methods may be partly dependent on the dataset being examined, and that Boosting may be very sensitive to at least one important aspect of a data set - noise. In fact, the results show that Boosting ensembles may overfit a concept in the presence of noise, greatly decreasing its performance. This work also suggests that most of the gain in performance for an ensemble comes in the first few classifiers combined, but with Boosting some further gains may be seen when combining up to 25 classifiers.
This work is joint work with Dave Opitz of University of Montana.