Methodological progress note: Pilot randomized controlled trials

Abstract

Definitive randomized controlled trials (RCTs) are the cornerstone of evidence-based medicine but can be complicated, protracted, and expensive. Given the challenges of large-scale trials, pilot trials serve as a crucial initial step, allowing for refinement and validation before embarking on the definitive RCT.1 They are a crucial element of good study design and, while conducting a pilot RCT does not guarantee success of the definitive RCT, it increases the likelihood of successful trial completion.2 More than US$100 billion is invested annually in biomedical research but often this research is conducted wastefully from poor study design and/or study procedures.3 Conducting a well-designed pilot RCT before launching an expensive, time-consuming definitive trial can minimize research waste and improve study conduct. Small RCTs cannot be branded pilot or feasibility trials to justify a small sample size. Pilot RCTs have a very specific purpose and inform future trial conduct.4 Indeed, research models, including the Canadian Critical Care Trials Group programmatic model, the UK Medical Research Council, and the Australian Clinical Trials Alliance, highlight the importance of pilot RCTs as an integral and necessary step in interventional clinical research (Figure 1). Early piloting of research methods and interventions is important in evaluating feasibility and acceptability before the definitive RCT. The importance of pilot trials has been acknowledged for decades5 with trial methods evolving over time. It is within this context that we will discuss pilot RCTs used to inform larger definitive RCTs. We will situate pilot trial methods within a larger research framework and propose important concepts in design and reporting. The terms “pilot” and “feasibility” trial are used interchangeably by some, but others purport that each type of trial has unique characteristics and therefore define them separately. Whitehead et al.4 proposed that pilot trials are a type of feasibility trial with some distinguishing elements: (i) stricter methodology (closely following the definitive study design); (ii) intended to lead to further work; (iii) a smaller version of the larger study; and (iv) focuses on trial processes. This delineation suggests that a pilot RCT is a specific subset of feasibility trial. Henceforth, we adopt the term “pilot.” Pilot RCTs allow researchers to test and establish feasibility of the study protocol, study processes, data collection, and intervention fidelity and acceptability.2, 4, 6 Table 1 details trial elements tested by a pilot RCT. Inclusion and exclusion criteria Participant retention Loss to follow-up Outcome testing Stakeholder engagement Statistical analysis plan Screening Recruitment Consent procedures Randomization Resource allocation Training requirements Tools/database Methods Follow-up plan Missing data Monitoring/quality assurance plan Blinding procedures Acceptability to clinicians and consumers Intervention refinement Protocol violations/deviations An important indicator of trial feasibility is the ability to recruit the required numbers of participants, using inclusion/exclusion criteria, from the sample population. Recruitment to RCTs can be challenging and investigators are frequently unable to achieve the predetermined participant numbers.6 Investigators' over-estimation of the size of the pool from which to sample ultimately leads to under-recruitment, negates the trial's ability to answer the proposed research question/s, wastes resources and can be unethical, and often results from foregoing a rigorous pilot trial process. Funders also value the importance of testing feasibility before embarking on definitive RCTs. For example, in criteria from the National Health and Medical Research Council of Australia, research projects must be “highly feasible with all required techniques and resources established.” Traditionally, pilot RCTs have been separate (i.e., external) to the definitive RCT. External pilot RCTs are a stand-alone trial completed before the definitive RCT: data is analyzed, results are generated, and thus inform feasibility of progressing to a definitive trial.7 External pilot RCTs are particularly useful if uncertainties exist around intervention fidelity or novel trial aspects (such as new clinical settings).8 However, data from external pilot RCTs are not included in the larger trial, published separately and therefore could be seen as “wasted.”4 Furthermore, attracting funding for external pilot RCTs may be difficult, particularly when study outcomes are unrelated to patient benefit, but rather focus on feasibility. Over the last decade, more researchers are using internal or embedded pilot RCTs as an innovative and efficient way to assess protocol and intervention feasibility and acceptability.9 Cooper et al. suggest this is useful when overall feasibility has largely been established, and evidence on recruitment, randomization, and attrition rates are required.10 Internal pilot RCTs are a phase within the definitive RCT and planning occurs alongside the main trial. Clear prespecified progression criteria (see upcoming section) are used to determine progression to a definitive RCT,9 with this decision ideally made by an external panel. If appropriate, and no significant changes are required to outcome definitions or interventions, participants' data from the internal pilot can be used in the final analysis of the definitive trial.8 Advantages include seamless integration into the definitive trial, ability to make real-time adjustments and efficiency through minimizing duplication and optimizing resources.9 Determining the sample size for pilot RCTs is not based on obtaining a sufficiently large sample so that between-group statistically significant (or nonsignificant) differences can be found. Pilot RCTs assess feasibility therefore sample size should reflect the ability to determine if trial processes and procedures are sound. Many methods of sample size determination have been proposed and most justify sample size through confidence intervals, means, and standard deviations. Estimates between 12 and 50 participants have been suggested,11, 12 whereas Stallard suggests the sample should be 0.03 times that of the definitive RCT.13 There is no consensus on the best method to use but, while it is not necessary for pilot RCTs to have formal sample size calculations, it is necessary for sample size justification to be given.4 For example, the pilot RCT14 in Table 2 calculated sample size at 110 participants (55/group) using methods from Hertzog.15 PICC insertion <18 years Predicted hospital stay ≥24 h Single lumen PICC Written informed consent by English-speaking legal parent/guardian PICC to be inserted for medical treatment Informed consent Vascular size sufficient to support 4-french PICC or larger Patients with a current bloodstream infection (<48 h) Vessel size 2 mm Could not speak English without interpreter Require a multilumen PICC Previous study enrollment Previous enrollment in current study Current catheter-related bloodstream infection Thrombosis in vein PICC is to be inserted Non-English speaking without an interpreter Known sensitivity to study products Admitted for COVID-19, or to a designated COVID-19 unit/facility Cook™ Polyurethane, turbo-ject, power-injectable PICC (standard care) BioFlo® polyrethan with Endexo® technology (intervention) Polyurethane PICC: pressure injectable polyurethane with external clamps (standard care) Hydrophobic PICC: with pressure activated valves; BioFlo (intervention) CHG-impregnated PICC: with external clamps; Arrow g+ard Blue Advance (intervention) >70% of patients will be eligible >70% will agree to enroll <15% will be lost to follow-up or withdraw from study >80% will receive allocated intervention <10% missing data PICC failure defined as the following complications: (1) CABSI; (2) Local site infection; (3) Venous thrombosis; occlusion; (4) fracture; (5) dislodgement. Parents (or caregivers)/staff satisfaction with study product (0–100 scale) Operators’ satisfaction with insertion Individual PICC complications Adverse events PICC dwell time PICC failure, a composite of thrombotic (venous thrombosis, breakage, and occlusion) and infective complications (PICC-associated BSI and local infection) severe enough to cease therapy All-cause PICC complications: a composite of the thrombotic and infective complications previously, but evident at any stage of PICC dwell, and may of may not require PICC removal Thrombotic complication: composite of venous thrombosis, occlusion, and breakage (any time during PICC dwell) Infection complication: composite of PICC-associated BSI and local infection (any time during PICC dwell) Individual complications: individual thrombotic and infective complications (any time during PICC dwell) Adverse events (allergic reactions, pain, and mortality) PICC dwell time Patient/parent satisfaction at insertion and study end (0–10 scale) Healthcare costs (direct product costs, healthcare resources, and failure-associated resource usage) Pilot trials prioritize logistics over statistics16 therefore, feasibility outcomes should be presented using simple descriptive statistics (counts, proportions), then compared against predetermined feasibility criteria.17 Clinical data collected as part of the pilot RCT can be analyzed using inferential statistical methods to demonstrate the soundness of the statistical analysis plan for the definitive trial so any necessary modifications can be made before the definitive trial commences. However, any between-group differences in clinical outcomes must be interpreted with caution due to the small sample. Eligibility (number of screened participants who will be eligible) Recruitment (number of participants who will provide consent) Protocol adherence (number of participants correctly receiving allocated treatment) Retention/attrition (number lost to follow-up/withdrawal) Missing data Staff/patient/consumer satisfaction and acceptability. Kleidon et al.14 use these criteria in their external pilot to determine the feasibility of a larger-scale RCT. In this case, all feasibility criteria were met, and the trial progressed to a definitive RCT18 (Table 2). Feasibility targets will be highly individual from study to study and must consider factors such as funding, number of recruiting sites, and incidence of the condition being studied. External and internal pilot RCTs require robust, predetermined progression criteria which guide if the pilot will “progress” to a definitive trial. It is imperative these are established before the pilot RCT begins to reduce bias, which may be introduced if progression criteria are set after results are known to investigators. The composite feasibility outcome measure above represents a way to determine progression to a definitive RCT. Different approaches are available to guide decision-making about the viability of undertaking a definitive trial. Thabane et al.6 describes four levels of interpretation of progression criteria: (i) stop the trial as not feasible; (ii) continue as feasible with modifications; (iii) continue as feasible with close monitoring; and (iv) continue as feasible without modification. Avery et al.19 explain a traffic light system to facilitate decision-making: stop/red (problems identified which cannot be resolved; further feasibility assessment required), amend/amber (issues potentially able to be remedied, proceed with caution; changes to trial processes and procedures required, with regular monitoring for effect) or continue/green (current processes support trial completion). Therefore, rather than strict “stop points” or barriers to trial progress, these criteria provide an opportunity to identify and rectify potential issues and ultimately ensure successful trial completion. In Herbert et al.'s20 informative paper on internal pilot RCTs, alternative real-world approaches to progression criteria are provided, in addition to an interesting discussion on the proportion of the definitive trial commonly used for the internal pilot. An example of a pilot RCT not progressing to a larger trial is a study from Marsh et al.21 Feasibility outcomes were set a priori (eligibility [>75%], consent [>90%], attrition [<5%]; protocol adherence [>90%]; and missing data [<5%]) and both eligibility and protocol adherence criteria were not met (16% and 88%, respectively), with eligibility identified as the greatest barrier to recruitment despite modification to inclusion/exclusion criteria mid-trial. Modifications and refinements to trial design, trial processes or intervention delivery are not uncommon before conducting a definitive RCT. Recruitment may be slower than expected so it may be necessary to adjust inclusion or exclusion criteria to achieve the required sample within the available timeframe and budget or it may be necessary to add additional sites in the definitive trial. If data collection is too onerous for trial staff, some secondary outcomes could be removed or data collection episodes reduced. The intervention may be complex and difficult to deliver therefore increased staff training may be required or education delivered differently. Care must be taken, however, with modifications to trial processes and procedures in internal pilot trials. Investigators (or external panel) must determine what degree of change is allowable, when retaining data from the internal pilot participants.8 Table 2 illustrates trial modifications required for a definitive RCT,18 informed by an external pilot RCT.14 Clear guidance on reporting pilot RCTs is provided in the Consolidated Standards of Reporting Trials (CONSORT) 2010 statement: extension to randomized pilot and feasibility trials1 which utilizes checklists for study abstract and body and a template flowchart to improve transparency and quality of reporting. The CONSORT statement mentions that, while it does not apply specifically to internal pilot RCTs, it may be applicable or modifiable in part for internal pilot RCTs. It is important to register pilot RCTs with a Clinical Trials Registry so study design is transparent before publication, thus reducing reporting bias.1 Publishing external pilot RCTs is an important step in dissemination of trial feasibility6 and has been aided by the launch of BioMed Central Pilot and Feasibility Studies (https://pilotfeasibilitystudies.biomedcentral.com/) whose core business includes publication of pilot and feasibility studies. For definitive RCTs with an internal pilot, feasibility outcomes from the pilot phase in addition to clinical outcomes (which include data from the internal pilot if no major modifications were made to population, intervention or outcomes) would be reported. The solid foundation provided by well-designed pilot trials is now broadly recognized. Pilot RCTs afford a preparatory phase for learning before expansion, mitigating the risk of logistical impracticalities leading to failure in subsequent large-scale RCTs. No large publicly funded RCT should be conducted without thorough piloting of interventions, study processes, and study procedures. Open access publishing facilitated by Griffith University, as part of the Wiley - Griffith University agreement via the Council of Australian University Librarians. Amanda Corley declares her employer, on her behalf, has received unrestricted investigator-initiated research grants, from Cardinal Health, 3M, and Eloquest (unrelated to the current project) and a consultancy payment from Wolters Kluwer for review of clinical practice guidelines. Nicole Marsh declares her employer has received on her behalf speaker fees from 3M and Medline; investigator-initiated research grants from Biolife, 3M, Eloquest, and Cardinal Health; and a consultancy payment from 3M (unrelated to the current project). Samantha Keogh declares her employer, on her behalf, has received monies from BD Medical and ITL Biomedical for Educational consultancies (unrelated to the current project).

Topics & Concepts