From ‘loose fitting’ to high-performance, uncertainty-aware brain-age modelling

Abstract

In brain-age modelling, a machine learning model is trained on a normative, usually healthy group of individuals to predict chronological age from neuroimaging data. This model is then applied to new data and the difference between predicted and chronological age—termed the brain-age gap (BAG)—is taken as a measure of deviation from ‘normal ageing’. This new area of research has generated large interest over the last decade and accelerated ageing has been associated with many different disorders and pathologies (Franke and Gaser, 2019). In contrast to this common practice, Bashyam et al. (2020) advocate the use of brain-age models with suboptimal performance (i.e. higher errors) rather than models optimized for predictive accuracy in a normative sample. The hope underlying this idea is that larger model errors might not be a random deviation, but a better approximation to biological age and thus more related to variables of interest. Considering that a perfect brain-age model would result in an error of zero, rendering it useless as random BAGs cannot be associated with any variable of interest, the authors’ suggestion to fit the model less accurately to generate larger, but not yet random model errors, seems intuitively reasonable. With their proposal the authors address fundamental, but thus far neglected implications of model performance in brain-age modelling. In particular, the large training sample size and the diversity of disorders investigated in their dataset provide a highly valuable contribution to the field. However, despite the intuitive appeal of ‘loose fitting’, suboptimal model training (i) violates the conceptual foundation of brain-age modelling; and (ii) is highly problematic from a methodological point of view and might lead to results that are uninterpretable. In the following, we first address the fundamental conceptual issue arising from ‘loose fitting’. Then, we provide evidence showing that, in contrast to the authors’ report, higher discriminative power for loosely fit models cannot be shown in their own data as well as under the most favourable of settings. Finally, we provide an analysis based on the decomposition of uncertainty components to show how suboptimal model fitting may lead to invalid results driven by increased epistemic uncertainty. To understand the conceptual problem of ‘loose fitting’, consider that brain-age modelling aims at learning the ‘normal’ trajectory of ageing by selecting a group of individuals whom we define as ‘normal’. The key to brain-age modelling—and normative modelling in general—is that normal ageing is defined as the trajectory observed in this normative (i.e. healthy) sample. Thus, fitting a more accurate, tighter predictive model directly translates into a more exact approximation of normal ageing. In turn, this enables us to interpret deviation from the model (i.e. larger model errors) as ‘non-normal ageing’. It follows logically that the case of zero model error outlined above (i) is not problematic, but desirable for the normative population; and (ii) can, by definition, not occur for individuals deviating from the normal trajectory of ageing as inferred from the normative sample. From this point of view, brain age modelling can be understood as an anomaly detection approach. The more accurately normative samples can be modelled, the more power these models have to detect anomalies (i.e. deviant ageing trajectories). It is also important to keep in mind that the selection process is a key to normative modelling. A randomly chosen population sample reflects the disease burden (prevalence of diseases of interest) in the general population, which might vary between regions and countries. Selection of a group of individuals without known diseases, so-called supercontrols, in contrast will enhance any difference in abnormality compared to a group of patients with diseases of interest. The authors justified the use of suboptimally fitted models with the assumption that the probed normative population (i.e. the healthy controls) was not in fact ‘normal’. Indeed, the authors’ argument that ‘the deep learning model might focus on imaging features and patterns that are not affected by pathologies, in an effort to match brain age and chronological age in individuals with such pathologies’ implies just this. Note, however, that in the framework of brain-age modelling, the algorithm—by definition—would never see an individual with ‘pathologies’ in the sense of a deviation from normality. Therefore, it could never make ‘an effort to match brain age and chronological age in individuals with such pathologies’. This shows that the authors’ line of argument can be sound only if the normative sample contains features affected by pathologies related to the deviation we later aim to discriminate. Conceptually, however, this is a direct violation of the normative principle underlying brain-age modelling as well as anomaly detection. In contrast to the authors, we thus conclude that the tighter the model fits the normative sample, the better we can discriminate deviants. Impact of ‘loose fitting’ (indicated by mean absolute error) on effect sizes of four group comparisons. Note that no systematic effect is evident. Impact of ‘loose fitting’ (indicated by mean absolute error) on effect sizes of four group comparisons. Note that no systematic effect is evident. As outlined above, the authors’ line of argument entails that loose fitting should be particularly beneficial if the normative sample contains features affected by pathologies related to the deviation we later aim to discriminate. Therefore, loose fitting, while not providing above-chance improvement in their own data, might still be helpful under conditions of high heterogeneity in the training set. To test this, we trained a neural network on n = 10 691 samples from the German National Cohort and applied it to a validation sample comprising n = 1986 individuals from the Marburg-Münster Affective Disorders Cohort Study (Vogelbacher et al., 2018). Note that the German National Cohort is intended as a population sample and does therefore contain a closer-to-normal spectrum of disorders and pathologies. If ‘loose fitting’ indeed increases discriminative power by avoiding fitting characteristics related to pathology, it should be most visible with this normative sample. Specifically, we tested healthy controls against four disorders, namely patients suffering from major depressive disorder (n = 822), bipolar disorder (n = 131), schizophrenia (n = 66), and schizoaffective disorder (n = 43) across six degrees of ‘loose fitting’ implemented by early stopping of neural network training controlling for age in analogy to Bashyam et al. We found no significant increase of discriminative power for any degree of ‘loose fitting’ in any of the four comparisons between healthy controls and patients (all z < 0.73, P > 0.307) with the impact of mean absolute error on effect size appearing rather random (Fig. 1). Now that we have established the lack of utility of ‘loose fitting’, we will consider its impact on model uncertainty. Note that BAGs are the sum of deviant ageing and model uncertainty, i.e. a person may have a large BAG not only due to actual changes in the brain, but also due to properties of the underlying machine learning model, which arise from characteristics of the training data such as data density and variability. Whether a given BAG should be considered a large deviation or a chance-level fluctuation is thus not only determined by the absolute deviation, but also by model uncertainty. Failing to properly model both epistemic and aleatory uncertainty may lead to spurious results which depend on properties of the model, which are produced by peculiarities in the training data rather than on the underlying association of a variable with brain-age difference (Marquand et al., 2016). Loose fitting specifically increases so-called epistemic uncertainty, i.e. uncertainty about the model weights (Kendall and Gal, 2017). Recognizing this issue, more recent publications aimed to explicitly model uncertainty (Palma et al., 2020); however, neglecting epistemic uncertainty. To show the impact of uncertainty, we trained a Monte Carlo composite quantile regression (MCCQR) neural network brain-age model on the German National Cohort sample and applied it to the MACS dataset (Vogelbacher et al., 2018). Figure 2 shows the effect of suboptimal fitting on epistemic uncertainty. Note that epistemic uncertainty as the percentage of total uncertainty decreases with longer training, rendering BAGs most confounded for the ‘loosely fitted’ model. Seemingly higher discriminative power might therefore arise from artificially increased epistemic uncertainty due to ‘loose fitting’. Epistemic uncertainty as percentage of overall uncertainty across epochs. Note that epistemic uncertainty is highest for ‘loosely fitted’ models and decreases as model training continues. Epistemic uncertainty as percentage of overall uncertainty across epochs. Note that epistemic uncertainty is highest for ‘loosely fitted’ models and decreases as model training continues. In summary, we addressed an important conceptual issue arising from ‘loose fitting’, precluding its use in an anomaly detection framework such as brain-age modelling. Next, we showed that, in contrast to the authors’ report, higher discriminative power for loosely fit models can neither be shown in their own data nor under the highly favourable circumstances of a heterogeneous normative sample. Finally, the analysis of epistemic uncertainty illustrates how suboptimal model fitting leads to higher epistemic uncertainty, resulting in BAGs confounded by data density and model weight distribution. To increase discriminative power, we thus suggest a careful selection of the normative sample and correction for aleatory and epistemic uncertainty in brain-age studies using algorithms capable of high-quality uncertainty estimation such as MCCQR regression. Data sharing is not applicable to this article as no new data were created or analysed in this study. This work was funded by the German Research Foundation (DFG grants HA7070/2-2, HA7070/3, HA7070/4 to T.H.) and the Interdisciplinary Center for Clinical Research (IZKF) of the medical faculty of Münster (grants Dan3/012/17 to U.D. and MzH 3/020/20 to T.H.). The analysis was conducted with data from the German National Cohort (GNC) (www.nako.de). The GNC is funded by the Federal Ministry of Education and Research (BMBF) [project funding reference numbers: 01ER1301A/B/C and 01ER1511D], the federal states and the Helmholtz Association with additional financial support by the participating universities and the institutes of the Leibniz Association. The MACS dataset used in this work is part of the German multicenter consortium ‘Neurobiology of Affective Disorders. A translational perspective on brain structure and function’, funded by the German Research Foundation (Deutsche Forschungsgemeinschaft DFG; Forschungsgruppe/Research Unit FOR2107). Principal investigators (PIs) with respective areas of responsibility in the FOR2107 consortium are: Work Package WP1, FOR2107/MACS cohort and brain imaging: Tilo Kircher (speaker FOR2107; DFG grant numbers KI 588/14-1, KI 588/14-2), Udo Dannlowski (co-speaker FOR2107; DA 1151/5-1, DA 1151/5-2), Axel Krug (KR 3822/5-1, KR 3822/7-2), Igor Nenadic (NE 2254/1-2), Carsten Konrad (KO 4291/3-1). WP2, animal phenotyping: Markus Wöhr (WO 1732/4-1, WO 1732/4-2), Rainer Schwarting (SCHW 559/14-1, SCHW 559/14-2). WP3, miRNA: Gerhard Schratt (SCHR 1136/3-1, 1136/3-2). WP4, immunology, mitochondriae: Judith Alferink (AL 1145/5-2), Carsten Culmsee (CU 43/9-1, CU 43/9-2), Holger Garn (GA 545/5-1, GA 545/7-2). WP5, genetics: Marcella Rietschel (RI 908/11-1, RI 908/11-2), Markus Nöthen (NO 246/10-1, NO 246/10-2), Stephanie Witt (WI 3439/3-1, WI 3439/3-2). WP6, multi method data analytics: Andreas Jansen (JA 1890/7-1, JA 1890/7-2), Tim Hahn (HA 7070/2-2), Bertram Müller-Myhsok (MU1315/8-2), Astrid Dempfle (DE 1614/3-1, DE 1614/3-2). CP1, biobank: Petra Pfefferle (PF 784/1-1, PF 784/1-2), Harald Renz (RE 737/20-1, 737/20-2). CP2, administration. Tilo Kircher (KI 588/15-1, KI 588/17-1), Udo Dannlowski (DA 1151/6-1), Carsten Konrad (KO 4291/4-1). Data access and responsibility: all PIs take responsibility for the integrity of the respective study data and their components. All authors and coauthors had full access to all study data. The FOR2107 cohort project (WP1) was approved by the Ethics Committees of the Medical Faculties, University of Marburg (AZ: 07/14) and University of Münster (AZ: 2014-422-b-S). The authors report no competing interests.

Topics & Concepts