Evaluation Methodology: ROC and FROC Curves

In mammography, the most common evaluation methodology is to compare the results obtained by the algorithms to those obtained by a set of experts, which is considered as ground-truth.

The performance of computer-based detection techniques can be summarized using Receiver Operating Characteristic (ROC) curves [120]. A ROC curve indicates the true positive rate (sensitivity) as a function of the false positive rate ($1-$ specificity). When no useful discrimination is achieved the true positive rate is similar to the false positive rate. As the accuracy increases, the ROC curve moves closer to the upper-left-hand corner, where a higher sensitivity corresponds to a lower false positive rate. A measure commonly derived from a ROC curve is the area under the curve $A_z$ , which is an indication for the overall performance of the observer [19]. For an ideal classifier the $A_z$ value is equal to one, or as a percentage (as used in the remainder of this thesis) equal to $100$ . It should be noted that ROC analysis is a pixel based assessment.

Region based analysis can be summarized using Free Response Operating Characteristic (FROC) curves [120]. This is similar to ROC analysis, except that the false positive rate on the x-axis is replaced by the number of false positives per image. In this case a definition of a detected region is needed and a typical approach expects a $50\%$ overlap between the annotated and detected regions to indicate a true positive. See also Appendix  for more information about both kinds of evaluations.