How robust are the binary classification performance metrics?

Accuracy, F1, or TPR are well-known and widely used metrics for evaluating and comparing the performance of machine learning based classification. But, are they or others such as BACC, CK, and MCC robust?

HIGHLIGHTS

• A benchmarking method is proposed for binary-classification performance metrics.

• Meta-metrics (metric about metric) and metric-space concepts are introduced.

• The method (BenchMetrics) tested 13 available and two recently proposed metrics.

• Critical issues are revealed in common metrics while MCC is the most robust one.

• Researchers should use MCC for performance evaluation, comparison, and reporting.

ABSTRACT

This paper proposes a systematic benchmarking method called BenchMetrics to analyze and compare the robustness of binary-classification performance metrics based on the confusion matrix for a crisp classifier. BenchMetrics, introducing new concepts such as meta-metrics (metrics about metrics) and metric-space, has been tested on fifteen well-known metrics including Balanced Accuracy, Normalized Mutual Information, Cohen's Kappa, and Matthews Correlation Coefficient (MCC), along with two recently proposed metrics, Optimized Precision and Index of Balanced Accuracy in the literature. The method formally presents a pseudo universal metric-space where all the permutations of confusion matrix elements yielding the same sample size are calculated. It evaluates the metrics and metric-spaces in a two-staged benchmark based on our proposed eighteen new criteria and finally ranks the metrics by aggregating the criteria results. The mathematical evaluation stage analyzes metrics' equations, specific confusion matrix variations, and corresponding metric-spaces. The second stage, including seven novel meta-metrics, evaluates the robustness aspects of metric-spaces. We interpreted each benchmarking result and comparatively assessed the effectiveness of BenchMetrics with the limited comparison studies in the literature. The results of BenchMetrics have demonstrated that widely used metrics have significant robustness issues, and MCC is the most robust and recommended metric for binary-classification performance evaluation.

Our article has been accepted by Neural Computing and Applications (SCI, Q1 journal) on May 2021

More info: ResearchGate | Full Text

Supplementary Materials and Data