A bootstrap approach is a superior statistical method for the comparison of non‐normal data with differing variances

Abstract

Phytologists have a primary interest in understanding plant growth, development and environmental responses. Experimentally, we rely on probing these by perturbing a system and monitoring changes, whether it be growth rate with temperature or gene expression in response to stress. To assess the significance of data from such experiments, frequentist statistics are used to ascertain the probability that a difference in a test statistic between conditions was due to chance (a P-value). When data are not normal, the adage is to use a nonparametric test for this analysis: the most common being the Mann–Whitney–Wilcoxon (MWW) test. Here, we explore conditions in which the MWW test is unsuitable and propose the use of a bootstrap approach instead. Most experiments aim to assess whether some metric is greater or less under different conditions, by an analysis of the change in the median or mean of that metric. The MWW test challenges the null hypothesis that two data distributions are the same (Mann & Whitney, 1947), not whether the two distributions have the same median. Therefore, it is possible to find a significant difference in an MWW test with distributions that have the same median, but different variances (i.e. heteroskedastic data) (Hart, 2001). These conditions are not uncommon in several experimental contexts, one being the measurement of symplastic cell-to-cell spread of green fluorescent protein (GFP) (Oparka et al., 1999; Burch-Smith & Zambryski, 2010). These assays allow the experimenter to count the number of cells (‘cell count’), or the number of concentric rings of cells (‘cell layers’), to which GFP has spread from a single cell (Fig. 1a) as a measure of the status of connectivity. Neither cell nor layer counts are normally distributed (Fig. 1b–d, upper), so most studies use the nonparametric MWW test to compare conditions to identify factors that regulate the connection and communication between cells. However, it is also clear that the shape of the distributions differs between experimentally compared conditions or genotypes (Fig. 1b,c) (Guseman et al., 2010; Diao et al., 2018; Cheval et al., 2020). Thus, if an MWW test is used on cell count data, the difference in distribution shapes between conditions may lead to the erroneous conclusion that there is a significant difference in the amount of spread of GFP. Therefore, a different statistical method is required to properly interpret differences in nonparametric heteroskedastic data. For this, we propose a bootstrap method (Efron, 1979). A P-value is defined as the probability of observing a value at least as extreme as the observed test static. This is done by a comparison with the null distribution, which describes the probability distribution of the test static when the null hypothesis is true. In the case of cell-to-cell connectivity, the test statistic is the difference in medians (). The null distribution describes the probability of observing a difference in medians when there is no true difference in the underlying data. Usually, a known distribution is used (e.g. t-distribution or F-distribution) but in this case it is unknown because the data do not follow parametric distributions (Fig. 1b,c). Bootstrapping techniques can be used to generate a null distribution de novo from the observed data already collected if the samples are independent. This removes the requirement of using a known distribution. To do this, the observed data are sampled with replacement to generate a resample. This mimics what the experimenter has done originally when observing the true population. The relationship between multiple resamples and the observed data can be used to reveal how the observed data relate to the true population, and so estimate a P-value for the observation. An example R function is provided to perform this analysis (Supporting Information Notes S1, medianBootstrap.R; https://github.com/faulknerfalcons/Johnston-2020-Bootstrap), which requires two arguments, that is two vectors of numbers: control and treatment. The function generates a null distribution to compare against by resampling each vector N times (by default 5000) and, for each resample, generates a resampled test statistic (). These N resampled test statistics are made into a null distribution by (Fig. 1b,c, lower) as suggested by Hall & Wilson (1991). where is the indicator function and is the nth resampled test statistic. As is an estimate of P a 95% confidence interval should be constructed, where P will fall within this range 95% of the time (Wilson, 1927). This method is not confounded by differences in variance or shape as with the MWW test. To illustrate this, we compared the type I error rate (false positives) between the MWW and medianBootstrap tests, when testing if there is a difference in medians between two populations for which there was no true difference in medians, that is (Fig. 2). In this scenario, an error rate of 5% is expected at . Equal samples () for each population were drawn from normal distributions with the same median and three different shapes, simulated in R v.4.0.0 (R Core Team, 2020). In the first instance, in which both distributions are equal (), both the MWW and medianBootstrap methods gave a difference in medians about 5% of the time, as expected (4.5% (95% CI [3.4, 6.0]) and 4.9% (95% CI [3.7, 6.4]), respectively) (Fig. 2a). When variances differed between populations (), ), the MWW test had a false-positive rate significantly higher than the set 5% of 7.5% (95% CI [6.0, 9.3]). Conversely, the false-positive rate of the medianBootstrap method was correctly controlled at 4.7% (95% CI [3.6, 6.2]) (Fig. 2b). When two samples are drawn from populations with equal variance and median, but differing shape and mean , a medianBootstrap method finds a significant difference in 5.1% of the trials (95% CI [3.9, 6.6]), as expected, whereas an MWW test inflates the type I error rate to 17% (95% CI [15, 19]) (Fig. 2c). The medianBootstrap method was robust to unequal sample sizes in all cases, provided the sample number was 10 or greater (Fig. 2). Thus, we recommend that both samples have at least 10 constituents for a robust result. Alongside this sample size requirement, samples must be independent and representative of the overall population. A bootstrapped difference in medians was used here as a comparison against the MWW test; it is worth noting that any test statistic, , can be computed in a bootstrapped manner, provided the test is invariant to scaling. This is required, as a bootstrap test generates the null distribution by rescaling the distributions around 0. Thus, the test statistic must be the same at X and X + n, for example the median and arithmetic mean are suitable, but the geometric mean is not. Therefore, the medianBootstrap method is a more appropriate analysis to identify differences in the spread of GFP, as cell count data exhibit unequal variances and differing distribution shapes between conditions and/or genotypes. Moreover, bootstrap testing can be extended to cell layer data, where means should be compared, as there is no difference in medians (Fig. 1d). An example of this extension is given in Notes S2, medianBootstrap.html. We acknowledge alternative advanced statistical techniques, such as linear mixed effects models, for analysis of these data. However, they require more assumptions and are less user friendly, often leading to mistakes (Knief & Forstmeier, 2020). In addition, we note that data are more complex than a single metric and support the movement to present figures with the data points, as well as summary statistics (Weissgerber et al., 2019). Moreover, while a summary statistic such as the median may remain unchanged, the distributions may still differ in a biologically relevant way, such as in a difference in variance. We consider this bootstrap method a good, easy-to-use, superior alternative to MWW analysis of cell-to-cell movement data. We thank Joanna Jennings (Department of Crop Genetics, John Innes Centre, UK) for providing the confocal micrograph in Fig. 1(a) and Joshua Hodgson (Department of Medicine, University of Cambridge, UK) and Matthew Castle (Department of Genetics, University of Cambridge, UK) for constructive comments on the manuscript. Data presented in Fig. 1 come from (b) Fig. S2 of Cheval et al. (2020), (c) Fig. 2(d) Diao et al. (2018) and (d) Fig. 2(c) Diao et al. (2018) under use of the CC-BY 4.0 licence. MGJ is funded by a John Innes Foundation Studentship. Research in the Faulkner laboratory is supported by the Biotechnology and Biological Research Council (BB/L000466/1, BBS/E/J/000PR9796) and the European Research Council (725459, ‘INTERCELLAR’). MGJ and CF designed, discussed and wrote up the research. MGJ performed the analysis. Notes S1 A short R script that provides a bootstrap function (medianBootstrap()) for the comparison of medians between two datasets, which returns a -value and associated confidence intervals. It can be used to directly replace the Mann–Whitney–Wilcoxon wilcox.test() function from base R. medianBootstrap(data1, data2, N = 5000, alpha = 0.05), requires two vectors of data (data1 and data2) and accepts to optional arguments: the number of bootstraps (N, default 5000) and the significance level for the constructing the confidence intervals (alpha, default 0.05). Notes S2 A primer on how to use the medianBootstrap.R code, with an example use case of the medianBootstrap() function; the code is then extended to replicate the functionality of a one-way ANOVA, allowing multiple-to-one bootstrap comparisons, as well as providing functions for the comparison of means and plotting the null distribution. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.

Topics & Concepts