Skip navigation
Full metadata record
DC FieldValueLanguage
dc.contributor.authorMcskassy, Sofus-
dc.contributor.authorProvost, Foster-
dc.contributor.authorRosset, Saharon-
dc.date.accessioned2005-11-02T21:03:33Z-
dc.date.available2005-11-02T21:03:33Z-
dc.date.issued2005-
dc.identifier.urihttp://hdl.handle.net/2451/14107-
dc.description.abstractThis paper is about constructing confidence bands around an ROC curve such that (1 - \delta)% of the ROC curves traced by data sets of size r will fall completely within the bands. We introduce to the machine learning community three methods from the medical field that are applicable to generate such bands. We then evaluate these methods on the simple case of “binormal” distributions— the scores for positive and the score for negative instances are drawn from two normal distributions. We show that none of the methods generate appropriate bands and investigate two types of variances problems. We show that widening the bands does not produce the proper bandwidths but that fitting a normal distribution to the observed drawn samples and drawing samples from this distribution (parametric bootstrap) does generate bands that are much closer to the desired coverage although still not perfect. We tested the original methods as well as parametric bootstrap on the covertype data set from the UCI ML-repority. The original methods perform the same as in the synthetic case, whereas the parametric bootstrap technique did not yield the expected results. This is primarily due to not being able to generate a good fit for the score distributions. Whether it is possible to fit well-behaving parametric distribution to learned models is an open question we leave to the machine learning community to answer.en
dc.format.extent285831 bytes-
dc.format.mimetypeapplication/pdf-
dc.languageEnglishEN
dc.language.isoen_US-
dc.publisherStern School of Business, New York Universityen
dc.relation.ispartofseriesCeDER-05-12-
dc.titleROC Confidence Bands: An Empirical Studyen
dc.typeWorking Paperen
dc.description.seriesInformation Systems Working Papers SeriesEN
Appears in Collections:CeDER Working Papers
IOMS: Information Systems Working Papers

Files in This Item:
File Description SizeFormat 
CeDER-05-12.pdf279.13 kBAdobe PDFView/Open


Items in FDA are protected by copyright, with all rights reserved, unless otherwise indicated.