The statistical properties of RCTs and a proposal for shrinkage
Erik W. van Zwet, Simon Schwab, Stephen Senn
Abstract
We abstract the concept of a randomized controlled trial as a triple (beta,b,s), where beta is the primary efficacy parameter, b the estimate, and s the standard error (s>0). If the parameter beta is either a difference of means, a log odds ratio or a log hazard ratio, then it is reasonable to assume that b is unbiased and normally distributed. This then allows us to estimate the joint distribution of the z-value z=b/s and the signal-to-noise ratio SNR=beta/s from a sample of pairs (bi,si). We have collected 23 551 such pairs from the Cochrane database. We note that there are many statistical quantities that depend on (beta,b,s) only through the pair (z,SNR). We start by determining the estimated distribution of the achieved power. In particular, we estimate the median achieved power to be only 13%. We also consider the exaggeration ratio which is the factor by which the magnitude of beta is overestimated. We find that if the estimate is just significant at the 5% level, we would expect it to overestimate the true effect by a factor of 1.7. This exaggeration is sometimes referred to as the winner's curse and it is undoubtedly to a considerable extent responsible for disappointing replication results. For this reason, we believe it is important to shrink the unbiased estimator, and we propose a method for doing so. We show that our shrinkage estimator successfully addresses the exaggeration. As an example, we re-analyze the ANDROMEDA-SHOCK trial.