ROC Confidence Bands: An Empirical Study

Abstract

This paper is about constructing confidence bands around an ROC curve such that (1 - \delta)% of the ROC curves traced by data sets of size r will fall completely within the bands. We introduce to the machine learning community three methods from the medical field that are applicable to generate such bands. We then evaluate these methods on the simple case of “binormal” distributions— the scores for positive and the score for negative instances are drawn from two normal distributions. We show that none of the methods generate appropriate bands and investigate two types of variances problems. We show that widening the bands does not produce the proper bandwidths but that fitting a normal distribution to the observed drawn samples and drawing samples from this distribution (parametric bootstrap) does generate bands that are much closer to the desired coverage although still not perfect. We tested the original methods as well as parametric bootstrap on the covertype data set from the UCI ML-repority. The original methods perform the same as in the synthetic case, whereas the parametric bootstrap technique did not yield the expected results. This is primarily due to not being able to generate a good fit for the score distributions. Whether it is possible to fit well-behaving parametric distribution to learned models is an open question we leave to the machine learning community to answer.