Receiver Operating Characteristic curve

Figure 2.1: Example of a ROC curve.

A good one-class classifier will have both a small fraction false negative as a small fraction false positive. Because the error on the target class can be estimated (relatively) well, it is assumed that for all one-class classifiers a threshold can be set beforehand on the target error. By varying this threshold, and measuring the error on the (maybe artificial) outlier objects, an Receiver Operating Characteristics curve (ROC-curve) is obtained. This curve shows how the fraction false positive varies for varying fraction false negative. The smaller these fractions are, the more this one-class classifier is to be preferred. Traditionally the fraction true positive is plotted versus the fraction false positive, as shown in figure 2.1.

Although the ROC curve gives a very good summary of the performance of a one-class classifier, it is hard to compare two ROC curves. One way to summarize a ROC-curve in a single number, is the Area-under-the-ROC-curve, AUC. This integrates the fraction false positive over varying thresholds (or equivalently, varying fraction false negative). Smaller values indicate a better separation between target and outlier objects.

Note that for the actual application of a one-class classifier a specific threshold (or fraction false negative) has to be chosen. That means, that only one point of the ROC-curve is used. It can therefore happen that for a specific threshold a one-class classifier with a higher AUC might be preferred over another classifier with a lower AUC. It just means that for that specific threshold, the fraction false positive is smaller for the first classifier than the second classifier.

In practical applications, the specific operation point on the ROC curve will not be known at the time of the training of the classifier. In many cases some range of reasonable false positives or false negatives can be given. It is therefore common to restrict the integration range for the AUC over this specific range. This will result in a more honest comparison between different classifiers for this application at hand. The toolbox therefore offers the possibility to compute the AUC over a limited integration range.