Popularity leads to bad habits: Alternatives to “the statistics” routine of significance, “alphabet soup” and dynamite plots

Abstract

Combine 3 with 1 in the presentation: Draw a bar chart of the means, with “SEM” error bars. Add stars/letters to each bar OR Table 1a and Figures 1a and 2a show typical examples of results presented using the Routine. These are fabricated, but one or other of these forms of presentation can be found in most issues of almost all biological journals, from the most humble to the most exalted. Other routines (Zuur & Ieno, 2016) are available, but are less widely followed. Example 1a also shows common features: the poor legend wording and no reference to the multiple range tests used (here, Duncan, 1955). This journal (AAB) has some reasonably strong guidelines for statistical presentation: These essentially prohibit use of the Routine. The roots of the Routine are hard to identify, but one of the seeds must surely be Ronald Fisher's book “Statistics for research workers” (Fisher, 1925), where tables of the values of statistics for particular p-values were first published, making statistical testing accessible. In conjunction with this, the last few decades have seen the widespread availability of software that makes the steps in the Routine very easy to carry out, with the consequent explosion in its use. Meanwhile, developments in statistical methodology have proceeded at exponential rates, driven by major developments in theory and computing resources. Whilst still including some significance testing, these newer techniques provide additional and more reliable information. Despite its widespread use, the Routine tends to mean that some useful information in the data is obscured, sometimes leading to unsupported conclusions. There is a very large body of literature, written over several decades, critically addressing the steps, generally suggesting alternative approaches that make more informative use of the data (Vail & Wilkinson, 2020). It would seem that no paper addresses all the steps in the Routine, with many of these papers not published in journals relevant to experimental biologists. This paper provides a partial review of publications relevant to all parts of the Routine. Informative data summary and data analysis are assisted by an understanding of ideas behind the methods used. Therefore, ideas relevant to the steps in the Routine are briefly summarised, along with the suggested alternative practices. The Routine results in a summary of statistical analyses; so understanding the purpose of a statistical analysis is a first step to gaining more information from data. Statistical analysis is about summarising data to find information and about making estimates and predictions (inferences) about what might happen in the future (Fisher, 1925). The need to deal with variation is inherent in these activities. If your dataset includes every possible data point, variation is not a problem because the dataset contains everything: the results of all football matches in a competition are known exactly. Very few research datasets contain all possible data points, so the key problem is to obtain representative summaries, estimates or predictions, and associated measures of uncertainty for these. Significance testing is a part of assessing uncertainty (Fisher, 1925), but has, in the view of many scientists, become synonymous with “the statistics,” with the consequence that potentially useful information is not found. An increased awareness of a fuller range of available analysis and data presentation tools can thus lead to an increase in information gained from a dataset. For many biologists, the “p-value” is the entire point of a statistical analysis. Some journals require a capital P, and even that it should be a capital, italicised P, perhaps reflecting this high importance attributed to p-values. AAB uses p, as do the majority of statistical and mathematical publications. There is a dichotomous interpretation of p: A significant “p-value” (usually one less than .05) is interpreted to mean that a difference between treatments is “real.” Conversely, if p > .05, then the treatments are “the same.” For a given comparison, it is also widely believed there is only one “true value” of p. These ideas are appealing, because they appear to give simple and easily interpretable conclusions. Sadly, the beliefs are mistaken (Goodman, 2008; Sterne, Cox, & Smith, 2001a; Wasserstein, Schirm, & Lazar, 2019). In the context of most significance tests, p simply stands for “probability,” and is: The probability of getting a result as extreme or more extreme than the one observed GIVEN THAT The null hypothesis of “no difference” is true AND THAT The assumptions behind the analysis carried out are (sufficiently) true. The last part is often not recognised, but it is very important. p-values can be small because the null hypothesis is not true, or because the result obtained is just one of the rare ones, or because the assumptions about the data required for a valid analysis are not appropriately satisfied. So, for a p-value to be meaningful, the data must sufficiently satisfy the assumptions: where this is not the case, a lack of adherence to assumptions can be the primary cause of a significant result. Statistical significance (or not) says nothing about biological importance (Ziliak & McCloskey, 2008): any “test” result must be interpreted in the context of the trial and relevant biology. Values other than .05 can be used to determine significance. Fisher (1925) promoted choosing a value to reflect the context (e.g., information from previous trials), so sometimes p= .1 might be a good choice, sometimes a value smaller than .05. Fisher suggested p = .05 was convenient, partly because, with the assumption of Normality, it approximately corresponds to a difference being equal to twice its standard error, partly because he thought a 1 in 20 chance was sufficiently small to be interesting (Fisher, 1925). Today, computers can calculate exact p-values in a fraction of a second, so these can be presented, allowing readers to make their own decisions as to the importance of effects (Webster, 2001). A natural interpretation is that the smaller the p-value, the more “evidence” there is that the effect is “real,” interpreting 1-p as “evidence against the null hypothesis.” Unfortunately, this is not the case. “Evidence against the null hypothesis” could be calculated using Bayes' theorem, but this requires further, usually unavailable, information (Matthews, 2001). In practice, it is reasonable to interpret a large value of p (close to 1) as evidence in support of the null hypothesis (no effect), and very small p values (such as p < .001) as evidence against the hypothesis (Sterne et al., 2001a). Ultimately though, results from many trials and sources are needed to confirm that an effect is “real.” Use of asterisks/stars is very common: the practice probably originated as a reference to a footnote referring readers to published tables of critical levels of a statistic. Stars convert a continuous 0 to 1 p-value scale into classes (usually, n.s., *, **, ***), losing both subtlety and information (Wasserstein et al., 2019) whilst simultaneously attributing undue accuracy to p-values near the cut-off points between the classes. It is more informative to present actual p-values, especially for n.s. (not significant), because n.s. can mean anything from a shade larger than .05 (slightly “interesting”) all the way up to p = 1 (identical means). p-values are only estimates: analysis assumptions are never exactly satisfied. Thus, very similar p-values, including p = .051 and p = .049, are essentially the “same” (Wasserstein et al., 2019) and should be interpreted as such. In general, p-values should be quoted to only two or three decimal places. One or two decimals are enough for associated statistics (F, t, etc.) if they are also presented. Trial design is fundamental to the value of a study (Finney, 1988), but is not mentioned in the Routine. Frequently, only the number of replicates and treatments used are provided (Haddaway & Verhoeven, 2015), perhaps because it is assumed that the design is always a randomised complete block design. There is a huge range of potential designs, old (Fisher, 1926) to recent (Williams & Piepho, 2019), and sound experimental design underpins effective and efficient studies (Finney, 1988; Kilkenny et al., 2009; Smith & Cullis, 2019). Appropriate analysis methods can vary substantially between different design types. An accurate and complete description of the design is therefore essential, to support the validity of trial results and enable reproducibility (Haddaway & Verhoeven, 2015). ANOVA is the most frequently used method for data summarised using the Routine. Like all statistical methods, ANOVA is underpinned by a set of assumptions that must be satisfied for the results to be valid. ANOVA has five underlying assumptions, with three being the most important (Finney, 1989). The analysis is assessing the differences (meanB−meanA) between treatments rather than other possible relationships (e.g., meanB being a multiple of meanA). The variance is the same for each treatment: Fisher (Fisher & Mackenzie, 1923) developed ANOVA with the primary aim to obtain a more robust estimate of the underlying variation (Finney, 1988). If the variation around each treatment mean is (close to) similar, a pooled (i.e., combined) error can be calculated, based on more data. The pooled error, and any test using it, is more reliable when individually calculated errors are used. Therefore, the pooled measure of error should be used in the presentation of analysis results (Welham, Gezan, Clark, & Mead, 2015, chap. 5). Any data point is independent of another data point. Lack of independence is often caused by taking multiple measurements of the same thing, either at different times (“Repeated measures”) or the same time (“Pseudo replication”). The analysis needs to be adjusted to allow for this, otherwise test results be frequently with p-values being The assumptions are less important. The data around each mean is the validity of the testing of the or not always the validity of the statistics (Finney, 1989). ANOVA is to from & it is not robust to variance et al., 2015). is when the result is because of experimental as the use of a where treatments have not (such can sometimes be need to be and only need to be approximately satisfied. is frequently to assessing Normality, or not assumptions at all & assumptions are not sufficiently many the problem by using statistical the two most common to of assumptions are to the data and then use the same analysis or to use a has ANOVA and standard are methods because the primary from the is estimates of means, techniques strong assumptions, require less assumptions to be satisfied. The most used methods (e.g., obtain of the and then carry out tests on rather than on the actual data. Thus, the primary from these methods is not estimates treatment means, but a test assessing the of the mean of the for p-values for these statistics usually be larger than for the same from a because more assumptions are for a analysis. Thus, in general, the test has more The of differences between mean not to differences between treatment needs to be when interpreting the results of a analysis. data to analysis was used to enable the assumptions behind standard ANOVA or to be sufficiently thus these methods to be used. The primary to data was a lack of variance and to allow analysis of data et al., 2015). In an with only computing this was the that could be reasonably methods that more the of the data are available and should be used in to data A key is that the results are then generally more interpretable from a biological point of A several that need to be when interpreting analysis These are with a If the data for each treatment are then the of the data is & For the variance with the of the treatment The variance of the is often ANOVA of the the of the of the when are For data points, the mean is the (i.e., to the of of the of all of the data The mean can be different from the The mean of is and the mean is = and differences between obtained on the scale can be as can or a significant difference a the difference between two that the are different at the required significance standard errors be ANOVA of the is the differences between the of the this difference the between the This not anything about differences between the when is: the was used for and for out of a These both the for the there is no way to the difference between the or and for the using ANOVA to these There are where makes as a for or for to a et an case, a give interpretable of the as as it to satisfy the assumptions of the analysis so that the are the data do not sufficiently to the assumptions, data and methods can generally be because methods are available that allow the of the data as to be methods are in some papers in many journals and are in many et al., 2015). & provide methods for the analysis of both and with interpretable the use of for is very but to for is just one of In this always to a For a are to the means, and the between and the is by the The is an part of a For a and a simple with one and associated to = Other useful analysis methods are for with and effects et al., 2015), or of a & If a dataset is using more than one a range of p-values be obtained for the same and can The p-value for a is by several including the of between treatments between means, of means, number of and analysis of the data in Table 2a test results can between different The data are for five replicates of two A and Table has the summary The treatments were using different but valid with p-values as The first test is a standard using the pooled The is the same but with the p-value calculated using a making the test partly In this all values of the data are many each time five data values to each The p-value then extreme the result with the actual data was to the results for all of the data. The test is the a of a This test the of the of the data & 1988). The test the treatments using a The includes a the to the means, to the of the being rather than the this is a in the hypothesis being The p-values vary from for the standard to for the The two tests have similar, but not p-values, are larger than for the this is because of the assumptions underlying the The interpretation of the test result with the because the treatments are being in different It would be to that the treatment vary because the result of the test was can that the values for treatment A were than for the of the treatment means, as in the not about the difference between the as test results can vary between different methods, so can errors and significance results from one analysis method are not generally to from another analysis methods need to be and the interpretation and presentation of results be by that Thus, it is usually not to results from different methods, including simple summary statistics from the results of a analysis. A trial or other of study has that determine the study is carried out & Mead, & The study lead to to be lead to the of to be treatments and treatment & The statistical analysis including tests at in be even the data are Thus, the need to each treatment with each other treatment should be very testing (or is a of the Routine, leading to the results are often not easily interpretable or a with treatments different of a The would be is the between and the of and is to be the possible of are different from each there is a large number of treatments (e.g., several it is often useful to make an test (e.g., in and then present the in of along with a measure of This can be the in This method can in the The aim for is often to find the few so testing every of treatments is often not Some are of more than (Finney, is the of the with the and of as as the one with the of are generally presented using Figures 1a and sometimes important in the data. they convert a continuous 0 to 1 p-value scale into two classes and not significant), losing even more information than the difference” “the also the key information between estimates of In if the were not the for would be is than and are reasonably The nothing they that is “the as but to is “the as It is more to an error as the as in For the make interpretation rather can and be different from whilst and are “the The effect seen can be a result of or are more The with all the treatment can be an analysis substantially between statistical some the only way to make the required be to do that use the standard error of a difference or an from this way give p-values to the same an it is to make treatment in and that the tests all should generally be as with a view to to be in future analysis it is most useful to on the of treatment in the especially of biological with actual p-values used as just the p-value or was is almost effects can be as interesting as significant so can be et using the A major for presentation of results is to There are many of the as to is most useful between show the Routine uses either individually calculated “SEM” or The is a & & & Wilkinson, and is the of the from the so the variation in the data For on of the data from the mean & & In the “SEM” is an and is a analysis (Vail & Wilkinson, 2020). It is a measure the mean of a to the mean of the underlying the validity of an “SEM” is on to analysis assumptions as to have any The “SEM” calculated individually for a treatment a of the variation in the because it is always smaller than the an estimate of variation or of a the “SEM” is as is it is calculated from a small number of values 3 or The is in the large associated the = and for 3 and ANOVA addresses this by information over the the pooled standard error has larger associated of and smaller so is to individually calculated the is the “SEM” so widely The would appear to what always that the “SEM” is smaller than the so it and a that the “SEM” can be easily used to two are and on need to be for “SEM” to allow of making rather In to the and there are many other of There are two & variation in the and of estimates as from an analysis. Appropriate to show vary between the two so the of tables and also The of error are widely 2009; & & & Table 3 the types. of error are in a or the error needs to be in the because interpretation of the information substantially between the error statistics are in the same way as error to but they are of error give different and so must make what error & In the use of to a measure of error is and can be & it a range that the can vary just so it should generally be In errors should be for use = = = It is frequently useful to the with an error & & because they give a measure of of the error, and can allow it to be into an alternative error (e.g., convert into In of data are summarised 1 of & The mean and are exactly the same for each of the = = Thus, the bar of the mean with are all The bar with “SEM” only because the errors are calculated individually for each so are where = the is driven by not by the variation in the data or are widely by the statistical & & Wilkinson, because they many of the in the data. The more presentation of the as with with the from and that the both in the of the data and the number of For both values of 1 is from a is is from a so is 3 is an set where the data are in two For a summary of the is the most and show of the information that can be seen in is large for presentation as in or as & be the standard not the seen for 3 in the some show that the “SEM” not reflect the variation in the it is more informative to present the in for present the and of the data or the range & & For or & & Wilkinson, et al., 2019) or use less than the standard bar chart with error whilst more information information to values are carried out with sound statistical software calculate estimates and errors associated with as a part of the because these is the primary to carry out the analysis. Thus, these estimates and errors should be presented, including when the estimates are to the In it can be to show statistics results from analysis. it is less to the two The of errors presented should be on the and of the the of the data and what are to In a on each mean be most If have carried out ANOVA on the the and an be especially where the is the same for all the for the examples in Table 1a and Figures 1a and 2a are in Table and Figures and The bar chart is the very many other different to data as a and the large presentation show et al., 2019). The of bar to be partly because that is what partly a lack of with other and partly a that a bar a in bar are both the and to Whilst bar can be a useful way to present other often be more three to bar are presented a a sometimes a and a is also a or The first two are generally most useful for the results from an and the for data these only the can easily be with can be obtained to the other Statistical software etc.) and software etc.) enable these other to be presentation and both in choosing what the should and the it, as and It is important to about what are to can be do not just the first out several different and different software the of available and some make it very hard to the a statistical as part of your this is not with an a research with rather than treatments that can this and treatment are of Use a sound trial the trial etc.) your data using summary statistics and out a sound statistical using an analysis method that as as is the of the data. estimates and associated measures of from that analysis the treatments as by their the and the research the trial design and statistical analysis If present data results from the estimates and associated measures of the informative parts of your data and analysis by including tables and types. that all parts of the p-values as evidence or interpret the the interesting of treatment in the data. Use of alternative to the standard Routine often more and for This is at partly because software makes using the Routine the to more informative practice are not some to more about statistical methods, and to use a range of The more and information for the same in data. There is a that is relevant to the in this This includes several of additional to in the were used to the ideas presented in this

Topics & Concepts