Fair evaluation of classifier predictive performance based on binary confusion matrix
Amalia Vanacore, Maria Sole Pellegrino, Armando Ciardiello
Abstract
Abstract Evaluating the ability of a classifier to make predictions on unseen data and increasing it by tweaking the learning algorithm are two of the main reasons motivating the evaluation of classifier predictive performance. In this study the behavior of Balanced $$AC_1$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>A</mml:mi> <mml:msub> <mml:mi>C</mml:mi> <mml:mn>1</mml:mn> </mml:msub> </mml:mrow> </mml:math> — a novel classifier accuracy measure — is investigated under different class imbalance conditions via a Monte Carlo simulation. The behavior of Balanced $$AC_1$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>A</mml:mi> <mml:msub> <mml:mi>C</mml:mi> <mml:mn>1</mml:mn> </mml:msub> </mml:mrow> </mml:math> is compared against that of several well-known performance measures based on binary confusion matrix. Study results reveal the suitability of Balanced $$AC_1$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>A</mml:mi> <mml:msub> <mml:mi>C</mml:mi> <mml:mn>1</mml:mn> </mml:msub> </mml:mrow> </mml:math> with both balanced and imbalanced data sets. A real example of the effects of class imbalance on the behavior of the investigated classifier performance measures is provided by comparing the performance of several machine learning algorithms in a churn prediction problem.