Representativeness Is Not Representative
Jonathan Huang
Abstract
From January 2019 to November 2020 alone, there were 1,432 PubMed-indexed publications citing the “UK Biobank” or “UKBB”; a resource linking extensive biomarker and phenotypic data from ~500,000 UK adults. The construction of big data repositories like the UKBB are phenomenal achievements of collective labor, and the rapid proliferation of papers is a testament to their potential. Since its public launch in 2012, however, epidemiologists and biostatisticians have raised concerns about inferential challenges presented by convenience sampling in the UKBB.1–5 These concerns apply equally to any voluntary-participation big data resource, where many unknown, and potentially unknowable, selection mechanisms synergize with high precision to threaten statistical inference.3,6 Surprisingly, however, there have been few efforts yet to quantify the extent of selection biases in the UKBB, doing so primarily by description2,7–9 or simulation,3,4 and none have demonstrated the results of proposed corrective actions. In this context, Stamatakis et al5 provide an important service by demonstrating the use of poststratification weights (i.e., standardization) as a simple and practicable solution to demographic nonrepresentativeness. Here, I review their findings with respect to representativeness, use two recent examples to illustrate why this framing misses important lessons on selection bias in typical UKBB studies, and conclude by emphasizing core analytic principles that continue to stand epidemiologists in good stead in the era of big data. IS REPRESENTATIVENESS EVEN THE GOAL? Responding to work by Batty et al9 which suggested that demographic nonrepresentativeness in the UKBB did not substantially affect estimates for well-known “risk factor” mortality associations, Stamatakis et al5 directly estimated changes in associations between several behavioral measures (smoking, alcohol use, physical activity, fruit and vegetable consumption, and a composite index) and mortality (all-cause, cardiovascular disease, and cancer) after standardization to an external reference.5 Weights were generated based on observed joint frequencies of six sociodemographic, behavioral, and outcome measures (categories of: age, sex, educational qualification, smoking, physical activity, and body mass index [BMI]) from the Health Survey for England (HSE), which is presumed to be nationally representative on the basis of probabilistic sampling and a nonresponse weighting strategy. Their Table 15 reweighting results reaffirmed the nonrepresentativeness of UKBB subjects: being older, better educated, more physically active, etc. than the HSE population. However, based on ratios of hazard ratios (RHRs; weighted/unweighted), standardization did not substantially alter evaluated associations (all within 15%) except the associations between no physical activity and cancer mortality (RHR = 1.21) and between alcohol use and cardiovascular mortality (e.g., previous versus never drinker RHR = 1.63). Qualitative interpretations were also largely unaffected except again for alcohol associations, which shifted from “protective” to null or harmful. Stamatakis et al5 were admirably judicious in highlighting vulnerabilities of their pragmatic approach, including a limit to the number and granularity of potential sources of nonrepresentativeness. On this basis, an optimistic reader might reasonably interpret that these findings vindicate Batty et al9 or that improving representativeness with a few more characteristics could fully resolve any remaining issues. After all, nonrepresentativeness is only a threat to inference insofar as sources of selection are not addressed.6 Unfortunately, because Stamatakis et al5 intentionally avoided discussing causal relationships that may underlie observed associations including mechanisms of selection, we readers are unable to assess the extent to which their standardization was effective in reducing bias. Lack of discussion of sources of bias prevents transparent evaluation of whether a sufficient set of compensatory variables have been included. The vague goal of restoring representativeness, then, obscures the more the specific and demanding objective of breaking statistical dependencies that arise from selection. To wit, we may be completely successful in reproducing the distribution of any number of observed characteristics in a population10 and still miss the most relevant sources of selection bias affecting a target relationship (Figure 1). In the worst case, making the sample representative of the external reference might be the wrong goal entirely, e.g., if the HSE is also subject to underlying causes of nonresponse not addressed by the weighting strategy. In fact, a closer look at Batty et al9 shows they found obesity associated with a reduced risk of cardiovascular mortality in the 2006–2008 HSE reference population (adjusted hazard ratio 0.72 [0.35, 1.47] in their Supplemental Figure 2) suggestive of potential collider bias—i.e., if sicker and overweight individuals tended not to participate. While Stamatakis et al5 did not examine this relationship, it is unlikely standardization to the HSE would have corrected the issue.FIGURE 1.: Residual selection bias due to unmeasured or incompletely standardized measures. Selection (S) may induce collider bias and backdoor paths between any number of unmeasured or incompletely measured (gray background) characteristics, biasing the effect of A on Y even after standardization and adjustment for certain observed (white background) factors. Graph made with causalfusion.net. SEP indicates socioeconomic position.What’s more, the idea that a single strategy for restoring representativeness may be a universal remedy to selection bias stands in contrast to lessons from recent studies conducted in the UKBB and other big data resources. In particular, biomarker and risk factor discovery studies have prompted a growing recognition of insidious, pervasive, and substantive selection biases that are amplified when big data are used for exploration. Here, I briefly review two pertinent examples: polygenic scores and coronavirus disease 2019 (COVID-19). POLYGENIC SCORES In early 2019, several papers demonstrated that genetic variation in UKBB participants were strongly associated with geography (e.g., Haworth et al11). One obvious explanation was families with certain genetic ancestries may be more or less likely to participate due to the requirement of living within 10 miles of an assessment center and that this and other participant characteristics varied by location. Moreover, even when individual genetic variants (single nucleotide polymorphisms) were not strongly associated with geography, the aggregation of many genetic variants into polygenic scores were. Coupled with the UKBB’s sample size, even substantial adjustments (technical variables, study center, and 40 genetic principal component scores) left stubbornly confounded associations between genotype and various outcomes such as income, education, and BMI.11 An earlier study in the Avon Longitudinal Study of Parents and Children (n = 7,508) demonstrated further indications of selection, with various polygenic scores associated with more (education score OR = 1.31, extraversion score OR = 1.06) or less (attention-deficit/hyperactivity disorder score OR = 0.84, depression score OR = 0.91, BMI score OR = 0.92) participation.7 Two obvious lessons arise here: First, the power afforded by these large population studies leads to amplification of subtle selection biases that are not easily corrected by adjustments for observed variables. Second, numerous antecedents to or descendants of causes of interest will influence selection into the UKBB and do so differentially. A combination of any two unmeasured factors would lead to residual selection bias (Figure 1). Moreover, how variables should be treated in analyses will depend on whether it is believed associations arise merely from collider bias, a consequence of the main exposure, or possibly both (Figure 2). In other words, routine strategies to restore covariate representativeness are unlikely to address all sources of selection.FIGURE 2.: Selection bias in genotype–outcome associations. Factors subject to collider bias via. selection (S) may be antecedents (ancestry, geography) or consequences (health, behaviors) of genotype and need to be treated accordingly in analyses, e.g., by conditioning or standardization, respectively. Graph made with causalfusion.net. SEP indicates socioeconomic position.COVID-19 SUSCEPTIBILITY Recently, efforts to search the UKBB and other datasets for factors influencing risk of COVID-19 infection have prompted researchers to highlight nonrandom selection into (severe acute respiratory syndrome coronavirus 2 [SARS-CoV-2] or antibody) testing as a major inferential threat.8,12–14 Selection into testing is pervasive: Under a 5% false discovery rate, Griffith et al13 found 32% (811/2,556) of evaluated UKBB characteristics were associated with testing status. In principle, the effect of any one of these factors on infection risk is susceptible to bias due to selection on any other cause. “Test-negative” approaches, restricting analyses to only individuals tested, may be effective insofar as the factors that influence testing, such as access, individual willingness, clinician preferences, etc., result in greater distortions than other sources of selection and approaches can be triangulated.14 Such approaches are particularly appealing for UKBB studies as they obviate the need for reference populations. Unfortunately, a comprehensive study among 14.7 million Canadians by Sundaram et al12 suggests test-negative designs may not well approximate COVID-19 risks in the general population: Using a (pseudo-) test-negative approach, individuals with many comorbidities (Aggregated Diagnostic Group score 7–27) appeared to have three-quarters the risk of infection than those with no comorbidities (adjusted odds ratio [aOR] = 0.75; 95% confidence interval [CI]: 0.69, 0.80), whereas the expected elevated risk was observed in the population (aOR = 2.13 [1.98, 2.29]). In the UKBB, the test-negative approach produced similar paradoxical associations with comorbidities (e.g., cancer OR = 0.77 [0.66, 0.91]) while sociodemographic associations were in expected directions (e.g., OR low education = 1.4, OR male sex = 1.23).8 One explanation is that a source of selection bias that causally precedes testing is more strongly associated with comorbidities than sociodemographics. For example, in a simplistic scenario where all testing occurs in a hospital setting, selection bias may already be present in hospitalized subjects and restricting to tested individuals may produce no benefit (Figure 3): e.g., inpatients with comorbidities may be less likely to have COVID-19 than new admissions. Alternatively, associations with sociodemographic or behavioral factors tangential to this selection mechanism may be less biased. In this case, restoring representativeness on the basis of sociodemographics would have little benefit. Moreover, even in this case where restoring population representativeness may be a desirable target, correcting for a single common source of selection bias provides no universal guarantees.FIGURE 3.: Selection bias in COVID-19 studies. Analytic methods that select on testing may be biased due to other dependences that are strongly related to the exposure of interest. Here hospitalization is a strong source of collider bias for morbidity-COVID-19 associations, whereas association between COVID-19 and SEP or sex may be less affected. Graph made with causalfusion.net. SEP indicates socioeconomic position.RECOMMENDATIONS AND CONCLUSIONS The overarching aspirations of UK Biobank studies are to leverage sample sizes and diverse biomarkers to accelerate discovery of individual and population disease mechanisms.15 These studies present more challenging scenarios of selection bias than most typical observational studies due to subtle population-level selection effects, high power, and searches over large numbers of exposures, biomarkers, and/or outcomes. In such settings, it may be particularly tempting to choose routines that can be quickly applied across all analyses such as a standard set of adjustment or standardization covariates. However, the relevant sources of selection will vary between studies and between different exposure–outcome associations within studies. One strategy or even one triangulation strategy14 is unlikely to be uniformly effective. There is no alternative to clear thinking about sources of selection bias for each relationship of interest, an effort that is stymied by vague concepts such as “risk factors” and “representativeness” and the high-dimensional discovery setting. For discovery settings, in particular, there is no guarantee any particular signal detected will be less affected by selection bias or that a failure to find a signal may not be a consequence thereof. There are no shortcuts: identifying the causes of selection biases are mandatory. Causal diagramming at a sufficient level of abstraction can help identify measured and unmeasured common causes and colliders that are shared across target associations for a given population. Once identified, many compensatory strategies exist including frequency weighting and propensity score weighting or adjustments (e.g., Heckman selection model)1 and quantitative bias analyses can be implemented.3,16 So, is representativeness important to UKBB studies? Ultimately, the epidemiologists’ credo: “it depends” does not fail here. Insofar as certain sociodemographic or behavioral characteristics used to define representativeness are the relevant sources of selection, they ought to be addressed and Stamatakis et al5 have shown one way to do so. However, representativeness is not necessary and far from sufficient10 in the UKBB context. Not only are there more opportunities for biases to arise, e.g., in the emergent properties of aggregated genetic variants, but also an inclination to overlook such biases when presented with a deluge of exposure–outcome associations. We are particularly vulnerable when results are precise and seemingly support intuitive conclusions. Investigators of small case–control studies would not be assured that representative demographics are necessarily indicative of an absence of selection bias. The same care should be taken with big data studies. ABOUT THE AUTHOR Jonathan Yinhao Huang is a Research Scientist at the Singapore Institute for Clinical Sciences. His research is on the application of analytic designs that best support causal inference in maternal and child health and human development. In particular, he has interests in the proper roles of molecular measures, the performance of novel estimators in realistic observational data structures, and the limits of evidence in policymaking.