Variance-based Active Learning

Abstract

For many supervised learning tasks, the cost of acquiring training data is dominated by the cost of class labeling. In this work, we explore active learning for class probability estimation (CPE). Active learning acquires data incrementally, using the model learned so far to help identify especially useful additional data for labeling. We present a new method for active learning, BootstrapLV, which chooses new data based on the variance in probability estimates from bootstrap samples. We then show empirically that the method reduces the number of data items that must be labeled, across a wide variety of data sets. We also compare Bootstrap-LV with Uncertainty Sampling, an existing active-learning method for maximizing classification accuracy, and show not only that BootstrapLV dominates for CPE but also that it is quite competitive even for accuracy maximization.