Please use this identifier to cite or link to this item: http://hdl.handle.net/2451/14107

 Title: ROC Confidence Bands: An Empirical Study Authors: Mcskassy, SofusProvost, FosterRosset, Saharon Issue Date: 2005 Publisher: Stern School of Business, New York University Series/Report no.: CeDER-05-12 Abstract: This paper is about constructing confidence bands around an ROC curve such that (1 - \delta)% of the ROC curves traced by data sets of size r will fall completely within the bands. We introduce to the machine learning community three methods from the medical field that are applicable to generate such bands. We then evaluate these methods on the simple case of ÃÂ¢ÃÂÃÂbinormalÃÂ¢ÃÂÃÂ distributionsÃÂ¢ÃÂÃÂ the scores for positive and the score for negative instances are drawn from two normal distributions. We show that none of the methods generate appropriate bands and investigate two types of variances problems. We show that widening the bands does not produce the proper bandwidths but that fitting a normal distribution to the observed drawn samples and drawing samples from this distribution (parametric bootstrap) does generate bands that are much closer to the desired coverage although still not perfect. We tested the original methods as well as parametric bootstrap on the covertype data set from the UCI ML-repority. The original methods perform the same as in the synthetic case, whereas the parametric bootstrap technique did not yield the expected results. This is primarily due to not being able to generate a good fit for the score distributions. Whether it is possible to fit well-behaving parametric distribution to learned models is an open question we leave to the machine learning community to answer. URI: http://hdl.handle.net/2451/14107 Appears in Collections: CeDER Working PapersIOMS: Information Systems Working Papers

Files in This Item:

File Description SizeFormat

 The contents of the FDA may be subject to copyright, be offered under a Creative Commons license, or be in the public domain. Please check items for rights statements. For information about NYU’s copyright policy, see http://www.nyu.edu/footer/copyright-and-fair-use.html Valid XHTML 1.0 | CSS