Measuring metacognition: A comprehensive assessment of current methods
Dobromir Rahnev
Abstract
One of the most important aspects of research on metacognition is the measurement of metacognitive ability. However, the properties of existing measures of metacognition have been mostly assumed rather than empirically established. Here I perform a comprehensive empirical assessment of many common measures (meta-d’, M-Ratio, M-Diff, AUC2, Gamma, Phi, and ΔConf), as well as two recent model-based measures (meta-noise and meta-uncertainty). I also develop novel Ratio and Diff variants for the measures AUC2, Gamma, Phi, and ΔConf, resulting in a total of 17 measures of metacognition. To assess the measures, I develop a new method of determining the validity and precision of a measure of metacognition. In addition, I examine each measure’s dependence on task performance, response bias, and metacognitive bias, as well as each measure’s split-half and test-retest reliabilities. Finally, I examine the influence of trial number. Reassuringly, I find that all measures of metacognition investigated here are valid and most show similar levels of precision. Another reassuring finding is that all measures have very high split-half reliabilities for trial numbers over 100. However, the test-retest reliabilities are often very low with important implications for individual differences research. Finally, most measures show only weak dependence on response and metacognitive bias but many measures are strongly dependent on task performance. This comprehensive assessment paints a complex picture: no measure of metacognition is perfect and depending on the details of the experiment, different measures may be preferable. Based on these results, I make specific recommendations about the use of different measures.