John D. Loeser Award Lecture: Size does matter, but it isn't everything: the challenge of modest treatment effects in chronic pain clinical trials

Abstract

1. Introduction Randomized clinical trials (RCTs) of treatments for pain have a long and distinguished history. The earliest clinical trials not only identified analgesic medications and their efficacious dosages but also contributed to the development of clinical trial research designs and methods that came to be used throughout medicine. The ground-breaking investigators who designed and conducted these early studies recognized that various sources of bias must be addressed,68,69,78,105 and appreciation of the fundamental roles of study design and statistical principles became widespread as experience conducting RCTs grew. In this article, we first present analyses of a sample of chronic pain trials that show a decline in treatment effect estimates over the past few decades and discuss the implications of these results for determining sample sizes for future chronic pain trials. We then review explanations for the failure of RCTs to demonstrate the efficacy of truly efficacious treatments and address the role of excessive placebo group improvement. Finally, we consider various approaches that have the potential to improve the informativeness of clinical trials and their assay sensitivity, that is, their ability to distinguish an effective treatment from a less effective or ineffective treatment. 2. “The greatest teacher, failure is”: falsely negative and inconclusive clinical trial results It has been recognized for at least 2 decades that clinical trials of psychiatric medications often fail to show a statistically significant difference between an active medication and placebo.29,53,63,74,82 Although some of these RCTs might have investigated treatments that truly lack efficacy, many were for medications that had demonstrated efficacy in multiple previous RCTs and had been approved by regulatory agencies around the world (eg, selective serotonin reuptake inhibitors for depression). Similarly, many RCTs of treatments for chronic pain have failed to demonstrate efficacy.19,22,31 Some of these results also might reflect a true lack of efficacy—either in general or for the specific dosage studied—but some RCTs have failed to show efficacy of medications at dosages that had demonstrated efficacy in previous trials, had been approved by multiple regulatory agencies, and are generally considered first-line treatments.19,22,31 It is common to refer to clinical trial results that fail to show the efficacy of truly efficacious treatments as “false negatives.” However, the failure of a clinical trial to reject the null hypothesis of no difference between an active treatment and placebo at a prespecified level of statistical significance does not necessarily indicate that the active treatment lacks efficacy.86 Such nonsignificant study results can be accompanied by confidence intervals that are consistent with the possibility of a clinically meaningful treatment effect. When there is such an outcome, the results of the trial should be considered “inconclusive” rather than “negative.”39 A failure to reject the null hypothesis can also be a result of chance, reflected in the type II error probability of failing to reject the null hypothesis of no difference between treatment groups when one truly exists. Table 1 presents a list of potential explanations for the failure of clinical trials of truly efficacious treatments to show their efficacy (see also Ref. 86). We focus on the roles of statistical power, excessive improvement in placebo groups, and various study methods and patient characteristics in contributing to falsely negative and inconclusive clinical trial outcomes. An additional explanation for such clinical trial results is the possibility that existing outcome measures have limited responsiveness to detect treatment effects. Most chronic pain RCTs have used numerical or visual analogue scales of pain intensity as primary outcome measures,101 but other measures that could serve as primary outcomes—for example, ratings of pain relief, global improvement, or disease-specific pain-related symptoms—might have greater responsiveness.18,44,97,98,102,109 Furthermore, chronic pain RCTs have typically not been designed to study patients selected on the basis of genotypes or phenotypes targeted by “precision” or “personalized” pain treatments. Although we believe that the development of improved clinical outcome assessments and of mechanism-based treatments16,25,100 may make important contributions to the identification of pain treatments with greater efficacy or safety, further discussion of these issues is beyond the scope of this article. Table 1 - Why can clinical trials of truly efficacious treatments fail to show their efficacy? 1. Chance 2. Placebo group patients improved “too much” 3. The optimal patients and phenotypes were not studied 4. Existing outcome measures have limited responsiveness to treatment effects 5. Temporal changes in characteristics of patients enrolling in trials 6. Temporal changes in types of clinical sites conducting trials 7. Research subject misbehaviour 8. Research site unintentional bias and misconduct 9. Inadequate sample sizes 3. Treatment effects and sample size determination Twenty years ago, Moore et al.79 concluded on the basis of a series of simulations that “size is everything” if the samples of patients enrolled in RCTs are to have adequate statistical power to provide credible estimates of the efficacy of acute pain treatments. The results of recent meta-analyses of assay sensitivity and placebo group changes in RCTs of chronic neuropathic pain have found that treatment effects have decreased and placebo group changes have increased over the past several decades, perhaps especially in the United States. Tuttle et al.110 concluded that from 1990 to 2013, placebo group changes increased while active treatment group changes remained relatively stable; as a consequence, “treatment advantage” vs placebo decreased substantially. Figure 1 presents the results of a second recent meta-analysis, on the basis of which Finnerup et al.32 concluded that, from 1982 to 2017, there was an increase in mean numbers-needed-to-treat (NNTs) that was associated with increases in placebo group change, study duration, and sample size (note that we refer to active and placebo group “changes” rather than “responses” because the term “responses” fails to encompass regression to the mean, spontaneous improvement, and other nonspecific sources of improvement or worsening that are not actual responses to active or placebo treatments).Figure 1.: Combined number-needed-to-treat (NNT) per year from a meta-analysis of randomized clinical trials of pharmacologic treatments for chronic neuropathic pain.32Although the results of these meta-analyses are generally consistent with what has been observed for RCTs in major depression13,53,106,119 and other therapeutic areas,5,52 only treatments for chronic neuropathic pain were examined and few such analyses have examined other chronic pain conditions.24 Nevertheless, the results suggest that factors such as increasing placebo group change and changes in study methods may be limiting or reducing estimates of the effects of chronic pain treatments, which would necessitate larger sample sizes for adequate statistical power to detect minimally clinically important effects. When planning a clinical trial, appropriate sample size determination is necessary to avoid exposing more patients than necessary to a potentially nonefficacious or harmful treatment, while also including a sufficient number of participants to demonstrate a true treatment effect, if one exists.26,77 Tuttle et al.110 presented differences between medications and placebo in the percentage decrease in pain intensity from baseline, and Finnerup et al.32 presented NNTs. Such data, however, are of limited value for determining sample sizes for analyses of continuous pain outcomes, for example, analysis of covariance adjusting for baseline pain, which is a common primary efficacy analysis used in confirmatory RCTs of chronic pain treatments.21 In addition to type I and type II error probabilities—typically prespecified as 5% and 10% to 20%, respectively—sample size calculations for continuous variables require specification of the magnitude of the treatment effect and the variability of the outcome measure. A well-accepted approach to sample size determination for such a primary efficacy analysis involves the standardized effect size (SES),26 which for a parallel group RCT is the mean change from baseline in the active group minus that in the placebo group divided by the pooled SD. 3.1. Methods and results We examined whether SESs of published neuropathic and non-neuropathic chronic pain trials have decreased over the past several decades by performing a secondary analysis of data from a recent meta-analysis of RCTs of efficacious medications conducted from 1980 to 2016 for low back pain, fibromyalgia, osteoarthritis pain, painful diabetic peripheral neuropathy, and postherpetic neuralgia.102 The purpose of the initial meta-analysis was to compare the responsiveness of ratings of average pain intensity (API) and worst pain intensity (WPI), and in the current analysis, we explored the trajectories of API and WPI SESs over time. Twenty-three articles were identified for inclusion, with publication dates from 1999 to 2013. SESs were extracted or calculated using other reported data, and positive values indicate that the treatment reduced API or WPI more than placebo.102 Mixed-effects meta-regression was used to test the significance of the relationship between time and both API SES and WPI SES. Preliminary analysis suggested that the relationships between time and both API SES and WPI SES were not linear. We therefore fit quadratic models regressing API SES and WPI SES on time and the square of time, where time is the number of years from 1999. Four articles included 2 active treatments compared with the same placebo arm. A robust variance estimator was used to account for correlations among the dependent effect size estimates in these 4 articles. All analyses were conducted using R version 3.5.1 with the robust.se function for robust variance estimation.46,47 Table 2 presents the parameter estimates for time and the square of time for the API SES and WPI SES models. Figure 2 shows that API SES and WPI SES both increased slightly for a short time, but on average, the slopes decreased for every additional year after 1999. These results are consistent with the results of the meta-analyses of neuropathic pain trials32,110 and demonstrate that the average benefit of efficacious analgesic medications shown in recent RCTs is modest. It is unknown whether the SESs for API and WPI will level off at approximately 0.30 or whether there will be a continued downward slope that will result in even lower SESs. Table 2 - Parameter estimates for the pain intensity models. Average pain intensity SES Worst pain intensity SES Estimate (95% CI) P Estimate (95% CI) P Intercept 0.379 (0.308 to 0.451) <0.0001 0.401 (0.344 to 0.457) <0.0001 Time 0.036 (−0.002 to 0.074) 0.07 0.027 (−0.007 to 0.060) 0.13 Time2 −0.003 (−0.007 to −0.0003) 0.04 −0.003 (−0.006 to −0.000) 0.06 Time is the number of years since 1999.CI, confidence interval; SES, standardized effect size. Figure 2.: Standardized effect sizes for average and worst pain intensity in randomized clinical trials of chronic pain treatments from 1999 to 2013.3.2. Implications The results of our analysis do not address the causes of the decline in the SESs found in RCTs of efficacious medications for chronic pain. It is possible to speculate that this decline is due to efforts by the scientific community and government regulators to increase the rigor of clinical trial design, execution, and analysis through methods such as comprehensive prespecification of study methodology and analysis, limiting multiple hypothesis testing unless proper statistical adjustments are used, and principled methods to accommodate missing data.41,42,50,51,99,103 Declines in SESs may also result from greater availability of pain treatments over time, which could reduce the pool of eligible patients and increase the percentage of study participants who have refractory pain.22 Given the evidence that expectations are a major source of placebo effects, it is also possible that placebo group changes increase as evidence for a treatment's efficacy accumulates and becomes publicly available.2 One important limitation of the present analyses is that they are based on published trials of 5 chronic conditions that reported both API and WPI. Although our results provide some information about the temporal trajectories of SESs from chronic pain trials, analyses that examine SESs for different chronic pain conditions or that include a larger sample of RCTs might produce different results; indeed, because clinical trials with nonsignificant results are less likely to be published, meta-analyses that include unpublished studies might show even greater declines. In addition, because the clinical trials we examined were limited to studies of efficacious medications for chronic pain, analyses of clinical trials of devices (eg, spinal cord stimulators) or of other nonpharmacologic treatments (eg, cognitive-behavior therapy and physical therapy) might also produce different results. For example, it has been observed that treatment effect estimates from RCTs of psychosocial treatments for depression are generally greater than those from trials of antidepressant medications; this observation may be explained by attenuation of the antidepressant treatment effect in trials in which a medication is compared with placebo and both groups are receiving intensive clinical management, which can be “substantially more therapeutic for patients with depression than doing nothing.”90 The mean SES of approximately 0.30 for the most recent published chronic pain trials mirrors the mean SESs reported in meta-analyses of efficacious antidepressants for major depression.43,61,62 Antidepressant trials share with analgesic RCTs several methodologic characteristics that might contribute to decreased assay sensitivity, including subjective outcomes, considerable placebo group improvements, and appreciable missing data.41,61,110 Given the consistent meta-analysis results, it is crucial that analgesic and antidepressant RCTs be designed with realistic treatment effect estimates. To detect an SES of 0.30 with 80% power (α = 0.05, 2 tailed) in a parallel group trial, at least 175 patients per group would need to be randomized. An SES of 0.30 can be considered a modest treatment effect, and its clinical importance will depend on the risks and benefits of the treatment and its clinical context.15,20 Such SESs reflect not only the specific effects of the treatments (eg, the pharmacologic activity of a medication) but also any methodologic characteristics of the clinical trials that decrease their assay sensitivity.19,22 In designing chronic pain RCTs, an SES of 0.30 can serve as a benchmark that could be considered when performing sample size determinations. This approach addresses both the modest apparent efficacy of existing treatments and any limitations of the clinical trial methods that have been used to study them. It is important to acknowledge, however, that it is usually recommended that sample size determination be based on specifying an effect size that would be of minimal clinical importance to patients, clinicians, and other stakeholders. Given the often poor tolerability and risks of many existing treatments, doing so might be challenging because even a minimal treatment effect could be considered meaningful for a novel treatment that is well tolerated and safe.15,20 4. Three eras of analgesic clinical trials The observation that clinical trials of medications with well-established efficacy are sometimes unable to demonstrate that efficacy provided the impetus for ongoing efforts to explain such results by examining associations between the research methods and patient characteristics of RCTs and their assay sensitivity. As can be seen from Figure 1, 3 eras of analgesic clinical trials can be identified from the NNTs associated with pharmacologic treatments for neuropathic pain.32 The first era—from the early 1980s through the early 1990s—has the lowest NNTs (ie, greatest treatment vs placebo differences) and consists primarily of relatively small cross-over trials conducted by investigators such as Mitchell Max, Michael Rowbotham and Howard Fields, Søren Sindrup, and Peter Watson. These studies were typically conducted at a single clinical site with patients who were either personally known by the researchers or carefully assessed by clinician investigators with substantial expertise. The second era—from the mid-1990s to the mid-2000s—reflects the involvement of pharmaceutical companies in developing drugs for chronic pain. The early clinical trials of gabapentin, duloxetine, and pregabalin were conducted at multiple sites but often included investigators at academic medical centers with experience treating or researching the specific pain condition being studied. The third era—from the late 2000s to the present—has the highest NNTs and includes multinational RCTs with large sample sizes using primarily for-profit clinical research centers that conduct clinical trials across a wide range of therapeutic areas. The decrease in treatment effects reflected in these increasing NNTs could be a result of changes over time in research methods, study sites, and/or the patients enrolled in the trials.32 Meta-analyses of RCTs of chronic neuropathic23,32 and musculoskeletal pain24 have found that greater trial assay sensitivity was associated with shorter trial durations and also smaller sample sizes. It is possible, however, that smaller trials that are negative or inconclusive are less likely to be published, and such publication bias might contribute to the results of these meta-analyses. Nevertheless, on the basis of data such as these, it has been suggested that larger and longer trials are not necessarily better at demonstrating whether a treatment is truly efficacious.72,88 The decreased treatment effects observed over the past several decades could be a result of the pharmaceutical industry conducting an increasing number of appropriately powered RCTs intended to fulfill regulatory requirements for study durations that can examine durability of treatment effects. In addition, analyses of RCTs of depression72 and Parkinson disease45 have suggested that effect sizes might be smaller for patients who are enrolled later in the trial than for those enrolled earlier, perhaps due to the enrollment of patients who do not fulfill eligibility criteria because of pressure on sites to complete enrollment requirements. Also, with longer trials—for example, durations of 12 weeks or more rather than 5 to 8 weeks—there may be greater placebo vs active group improvement resulting from, as discussed in the next section, a greater number of study visits90 and an increased opportunity for patients to develop supportive relationships with study staff.87,91 It is also possible that over the course of these 3 eras of analgesic trials, the quality of RCT procedures and data, including patient clinical evaluations and outcome assessments, became more variable as greater numbers of study sites participated.74 In addition, there has been increasing recognition of the potential roles of unintentional and intentional investigator bias64,67,81 and frank research misconduct27 in contributing to negative, inconclusive, and invalid study results. It has also become apparent that surprisingly large percentages of the participants enrolled in clinical trials are either professional subjects who are fabricating a clinical condition—and may be participating in more than one clinical trial at different sites, so-called “duplicate patients”—or are patients who intentionally falsify key eligibility criteria to be randomized.10,11,76,96 Information provided on social media71 and clinical trial websites can facilitate enrollment of such unqualified participants, and methods to identify professional subjects and mitigate patient misbehavior are now being developed, including the creation of research subject registries.76,96 5. Placebo group changes and their interpretation The results of meta-analyses of RCTs have found meaningful relationships between placebo group changes and study methods and patient characteristics. Paralleling the results discussed above for treatment effects, greater placebo group changes in neuropathic pain trials were associated with longer trial durations and larger sample sizes.19,32,110 In a larger number of meta-analyses of major depression trials, greater placebo group changes were associated with larger numbers of study sites, larger samples, greater frequency of study visits, longer trials, lower probability of receiving placebo, and higher patient expectations for improvement.29,35,84,87,92,111,118 A robust finding that has emerged from multiple analyses of both pain and psychiatric treatments are associations between greater magnitudes of placebo group change and negative or inconclusive clinical trial outcomes, as evaluated, for example, by statistical and In such it is important to that in the magnitudes of placebo group change across a of RCTs will an between placebo group changes and treatment effect estimates that reflect the difference between that placebo group and an active treatment. As observed many years ago, a between and placebo in clinical trials does not of indicate the of a of Such an effect is to be on statistical and there is no need to for medical the statistical basis of associations between placebo group changes and treatment effect these associations can also reflect characteristics of the clinical trials that potentially reduce assay sensitivity. For example, it is for the mean pain intensity to a mean of 3 or 4 on a to numerical Such a of may an of refractory pain that if by patients in the placebo group would make it to show any further pain from an efficacious treatment. this effect it could at least in for the associations between greater magnitudes of placebo group change and decreased treatment effects that have been that there is such a effect, the between an efficacious treatment and placebo in an RCT might be greater if nonspecific sources of improvement in both treatment as placebo effects and regression to the be which could make it less likely that the placebo group would the explanation for associations between placebo group changes and treatment effect estimates involves the of in clinical trials. It is generally that the specific effects of an active treatment provide an benefit to the nonspecific effects associated with treatment in the placebo which include placebo effects and regression to the As by this that the active results from a placebo effect and that the placebo effect in the active is to the placebo effect of the it is possible that to the active treatment at least of the placebo group in which the specific effects of the active treatment and the effects of trial including placebo treatment, would be that is, some of the nonspecific effects that in the placebo group would not in the active treatment An of such is provided by et who that therapeutic with study have been reported to in what they consider appropriate with study be to improvement in patients with patients in the placebo of antidepressant In several trials, number of study was more associated with improvement in the placebo groups than in the antidepressant It was concluded that the number of study increases placebo while medication generally for example, only rather than over the course of a trial was associated with a difference in between an antidepressant and placebo of vs Such effects on active and placebo group if could reduce the apparent benefit of an efficacious treatment when compared with Although this would decrease the assay sensitivity of any trials in which it it does provide a basis for that assay sensitivity can be increased if study procedures such as or study have effects on active and placebo group 6. in is the approaches to the design of pain clinical trials does when determining the number of participants for an RCT to provide adequate statistical power to identify minimally clinically important and to their Nevertheless, it is important to that various for increasing the assay sensitivity of RCTs and the probability of inconclusive results should also be methodologic As in the on on in clinical trials, the in designing clinical trials is to identify the scientific of and the of the treatment effect the clinical by the trial The of the clinical trial design and the statistical analysis including methods for and missing data and the and interpretation of sensitivity of the and statistical issues in determining and principled statistical analyses for their is beyond the scope of this however, we believe it is important to that with in clinical trials should be from the earliest of conducting a clinical trial and through its design, execution, analysis, and The evidence that of clinical trial eligibility criteria can to intentional and unintentional among study and potential participants has provided a basis for that key of the that do not should be from study and and patients to eligibility criteria could reduce the numbers of patients who are randomized but who do not fulfill these criteria because of or baseline of and has of such relatively In addition, study and patients to when patients are more likely to be randomized to active vs placebo treatment (eg, finding and active could also the increases in placebo group that have been found in trials in which patients that their of receiving placebo is less than their of receiving an active treatment as a have greater expectations for patients and to the considerable to the used in and patient and also involves to the benefits on assay sensitivity that might An important of clinical trials that is receiving increased as a source of poor data quality and of to demonstrate the efficacy of truly efficacious treatments is poor treatment medication can decrease estimates of efficacy and assessments of but it has typically been assessed using which are known to be Although there are now a of more methods for medication that have greater they have been used in chronic pain placebo group changes also has the potential to assay sensitivity. For example, in neuropathic pain RCTs, the time to of pain in placebo groups has been shown to be longer than that associated with analgesic This is consistent with the observation that longer trials to have a increase in placebo group and that shorter treatment durations may be for of RCTs with longer durations would be necessary to the durability of any In addition, when potential participants for a clinical trial

Topics & Concepts