Publications, replication and statistics in physiology plus two neglected curves

Abstract

I was asked to write a short editorial to reflect on my links to The Journal of Physiology and to consider some of the changes in physiology reflected in The Journal. The pandemic has delayed my efforts just as it has delayed many planned gatherings of scientists. As physiologists, we all view physiology in different ways, depending on what levels of function we try to unravel – from the level of molecules and their interactions to the level of populations of cells and neurones, and finally to the level of whole organisms. Across our scientific careers, most of us stick with one of those levels and we likely over-estimate the importance of our own contributions. My first publication in The Journal was in 1976 – my Bachelor of Medical Science thesis on proprioception (Gandevia & McCloskey, 1976). This has remained a theme of my work ever since and my colleagues and I continue to explore new high-level proprioceptive illusions affecting the hand (Heroux et al. 2018). However, serendipity delivered another theme because I was not able to begin my proprioceptive experiments on time in 1975 because the equipment had not been finished. So, my supervisor Ian McCloskey and I turned to study the effect of activation of proprioceptive afferents with chest vibration on breathing patterns in anaesthetised rabbits. The work was duly published in 1976 and after graduating in Medicine, I continued my clinical neurophysiological interest in the neural control of respiration, exclusively in humans thereafter. This dual start meant that I was comfortable to approach broader questions about the control of voluntary movement. This led, for example, to work on the control of the human diaphragm (e.g. Gandevia & Rothwell, 1987) and upper airway muscles (Saboisky et al. 2007) and broader work on central fatigue (e.g. Gandevia et al. 1996; Taylor et al. 1996; Gandevia, 2001). Nonetheless, I was surprised in 2011 when I realised, I had published over a hundred papers in The Journal, and became an honorary member of The Journal of Physiology’s Century Citation Club (https://physoc.onlinelibrary.wiley.com/hub/journal/14697793/features/century-citation-club). Subsequently, I was asked by the editor of the Physiological Society magazine how I had reached this milestone. Apart from the inevitable passage of time, it indicated my simultaneous pursuit of several lines of physiological investigation. Why did I continue to publish in The Journal? My reply was that I believed that submitted papers nearly always received a fair appraisal based on good-quality reviews. Formal inspection of issues of The Journal published in 1976 and then at five-year intervals thereafter reveals many changes1. Most notable is the growth in the number of authors per paper from a mean of 2.2 ± 0.9 (SD; n = 86) in 1976 to 5.1 ± 2.9 (n = 103) in 2016. Less obvious is the increase in the size of papers from around 9000 words to 14,0002. One explanation is the move to bigger teams involved in projects along with the use of more techniques. Second, the proportion of papers with an overt clinical link (such as studies in humans with disease or animal models) has increased – this translational focus is a trend promoted actively by the previous and current Editors-in-Chief of The Journal. Third, the development of molecular and imaging techniques has provided many new and sometimes unexpected physiological landscapes. A personal example. In my work with colleagues on the upper airway we manipulated a magnetic resonance imaging method to view the motion of the largest airway dilator muscle in humans, the genioglossus. Surprisingly, we found genioglossus moves forward about half a millimetre with each inspiration in quiet breathing (Cheng et al. 2008), a finding later corroborated with ultrasound (Kwan et al. 2014). This movement is linked to the cross-sectional area of the upper airway. Not surprisingly, it moves further in those with a narrow airway. The new vista was exposed by exploratory research with new techniques rather than de novo ideas. The distinguished physiologist and geneticist J. B. S. Haldane felt that while new ideas were obvious to onlookers, new techniques were more important. Recently The Journal adopted a more stringent approach to the presentation of data and statistical reporting (Forsythe et al. 2019). Why was this needed? In part it stemmed from a realisation that many if not a majority of research findings are likely to be false (Ioannidis, 2005), and thus many publications were inevitably contributing to what some call a crisis in reproducibility (Baker, 2015) (cf. Fanelli, 2018). In addition, my colleagues and I had shown that guidelines for authors in the form of multiple editorials published in 2011 in The Journal of Physiology and the British Journal of Pharmacology failed to improve the standard of presentation of data and the reporting statistical findings in either journal (Diong et al. 2018). As examples, about 80% of papers mistakenly used standard errors of the mean as estimates of variability (rather than of uncertainty), and in more than half the instances when exact P values were given between 0.05 and 0.10, they were reported as trends or statistically important. We look forward to auditing papers in The Journal again to plot the improvement in standards now that The Journal mandates rather than suggests adherence to its statistics policy. Not only was I enlightened by the analysis of John Ioannidis of the landscape of irreproducible results but also by the work of Kahneman3 and Tversky into fundamental cognitive illusions (e.g. Tversky & Kahneman, 1974). Much of the latter work is summarised in Kahneman's bestseller Thinking Fast and Slow (2011). As a result, I teach an annual course in cognitive illusions for scientists. But what to do about the high risk of false positive results? One seemingly attractive way to improve the robustness of experimental findings is to adopt a more stringent threshold for statistical significance – traditionally 0.05. Another is to abandon the use of null hypothesis significance testing (NHST) altogether (Amrhein et al. 2019), and replace it with presentation of effect sizes and confidence intervals (ESCI). This view would avoid the phoney dichotomy of significant vs. insignificant and is promoted by Cumming and colleagues (e.g. Cumming & Calin-Jageman, 2016) and now some journal editors (Bernard, 2019). Papers using only this approach have already appeared in The Journal of Physiology (e.g. Heroux et al. 2018). Finally, I want to make two points about a depressingly little-known relationship between the P value obtained in an initial experiment and the probability that it will be reproduced (at a threshold of 0.05) in a direct attempt at replication (Fig. 1). It should be as well known as the haemoglobin dissociation curve. First, the curve shows (filled circles), not surprisingly, that as the initial P value gets small, say below about 0.01, the probability of a successful replication (P < 0.05) rises, reaching more than 90% for initial P values below 0.001 (Goodman, 1992). Second, it demolishes the allure of an initial P value around the traditional value in biology of 0.05. Here the chance of a successful replication is around 50:50 – the flip of a coin! How many appreciate the real significance of this? Because of the internal and external pressures to find and publish new things, some talk up and put ‘spin’ on the importance of an experimental result with a P value of say 0.055. However, I have never seen anyone say such a result had a trend to be statistically insignificant or was of borderline insignificance. Unfortunately, P values around 0.05 do not signify a trend (Wood et al. 2014). Still there is a further cautionary tale about replicability, even for studies with an initial P value around 0.001. This is because P values, even if very small, dance around, and they are a poor representation of the size of any potential effect (e.g. Halsey et al. 2015; Cumming & Calin-Jageman, 2016). I am grateful to Geoff Cumming for pointing out the numbers. The 80% prediction interval, i.e. the interval that includes the P value given by replication with an 80% chance, is staggeringly wide – for an initial P value of 0.001, the interval is 0.0000002–0.070; and for an initial P value of 0.05 it is 0.00008–0.44 (Cumming, 2008, see Table 1; values for a one-tailed replication P and a two-sided P interval). The message here is that even low initial P values will frequently fail the replication test. A P value and the truth can be too easily parted! As physiologists seeking the truth, we need in our toolkit convergent lines of experimental evidence from multiple studies along with direct replication. In his discovery of the circulation, don't forget that William Harvey relied on ‘both argument and ocular demonstration’. Hopefully you will be convinced to publish in The Journal of Physiology given it is keeping up with the overdue changes in scientific reporting and statistics, as well as ethical issues related to human and animal studies.

Topics & Concepts