Abstract
Background Adaptive design methods are intended to improve efficiency of clinical trials and are relevant to evaluating interventions in dialysis populations. We sought to quantify the use of adaptive designs in dialysis clinical trials.
Methods We completed a full text systematic review and adhered to the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) guidelines. Our review utilised a machine learning classifier and a novel full text systematic review method. We searched MEDLINE (Pubmed) and performed a detailed data extraction of trial characteristics and a completed a narrative synthesis of the data.
Results 50 studies, available as 66 articles, were included after full text review. 31 studies were conducted in a dialysis population and 19 studies had renal replacement therapy as a primary or secondary outcome. While the absolute number of adaptive design methods is increasing over time, the relative use of adaptive design methods in dialysis trials is decreasing over time (6.1% in 2009 to 0.3% in 2019). Adaptive design methods impacted 52% of dialysis trials they were used in. Group sequential designs were the most common type of adaptive design method used. Acute Kidney Injury (AKI) was studied in 27 trails (54%), End Stage Kidney Disease (ESKD) was studied in 22 trials (44%) and Chronic Kidney Disease (CKD) was studied in 1 trial (2%). 26 studies (52%) were supported by public funding. 41 studies (82%) did not report their adaptive design method in the title or abstract and would not be detected by a standard systematic.
Conclusions Adaptive design methods are employed in dialysis trials, but there has been a decline in their relative use over time.
Registration Number PROSPERO: CRD42020163946
What was previously known about the specific topic of the manuscript?The use of adaptive designs methods in dialysis trials is unquantified.
What were the most important findings? If studies are animals, this should be specified Although absolute numbers of adaptive design trials have increased over time, the proportion of dialysis trials using an adaptive design has reduced. Among trials that employed an adaptive design, 52% of dialysis trials were revised due to the adaptive criteria. Group sequential designs were the most common type of adaptive design method used in dialysis randomized clinical trials. Acute Kidney Injury (AKI) was studied in 54% of trials and End Stage Kidney Disease (ESKD) was studied in 44% of trials, which used an adaptive design.
How does the new information advance a new understanding of the kidney and its diseases?Adaptive design methods are effective in dialysis trials, but their relative use has declined over time.
Introduction
Background
Randomized clinical trials are the gold standard for evaluating efficacy, futility or harm of new therapies (1). Compared to similar specialties, nephrology has traditionally had a low number of randomized clinical trials, particularly evident for patients with End Stage Kidney Disease (ESKD) (2). The comparatively low number of trials are postulated to be due to difficult recruitment, previous history of underpowered trials and lack of funding (3,4). Although the number of trials are increasing, nephrology continues to lag behind other specialities such as cardiology, haematology/oncology and gastroenterology (5,6).
Adaptive clinical trials use interim data analyses to modify the trial design or duration in a predefined way (7), without undermining the integrity or validity of the trial, thereby preserving the Type 1 Error (False Positive) rate. The most common type of adaptive design is the Group Sequential Design (GSD), where planned interim analyses permit stopping of trials for efficacy or futility. Other designs include sample size re-estimation, multi-arm multi-stage trials, adaptive randomisation, biomarker adaptive and seamless phase II/III trials (8).
Adaptive clinical trials appear particularly suitable for the evaluation of novel interventions in ESKD, by reducing resource requirements, decreasing time to study completion and increasing the likelihood of study success i.e. power to answer hypothesis (9). Previous trials in ESKD have overly relied on observational data to inform trial design, including assumptions of expected effect size and variance (10), rather than estimates from early phase clinical trials. If incorrect, trials may be underpowered with an insufficient sample size to answer the underlying research question (10). Adaptive sample size re-estimation is a potential solution, as seen in cardiology trials (11), such as planned blinded sample size re-estimation, which identifies inaccurate assumptions, thereby triggering altered recruitment targets mid trial to ensure adequate power.
Adaptive design may also be relevant to evaluation of more established interventions. For example, the 4D trial (12) reported that atorvastatin 20mg per day did not reduce cardiovascular events in ESKD despite evidence of a 20-30% reduction in other populations (13). This trial included a single dose of statin or a single low-density lipoprotein target; it is hypothesised that alternative or multiple doses may have been more beneficial in an ESKD population given the significantly altered pharmacokinetics and pharmacodynamics (10,14). An adaptive multi-arm multi-stage (MAMS) trial design may have been more appropriate with one interim analysis at the end of stage I to identify an optimum dose to take forward into stage II. For example, the Telmisartan and Insulin Resistance in HIV (TAILoR) trial used a MAMS design with one interim analysis to identify the most appropriate dose among three telmisartan doses (20, 40 and 80mg daily). All three doses were tested in stage I and telmisartan 80mg was taken forward into stage II (15).
This systematic review aims to: (i) summarise the use of adaptive design methodology in randomized clinical trials in dialysis population and populations at risk of requiring dialysis; (ii) describe the characteristics of the adaptive designs including dialysis modality, funding, and geographical location; (iii) describe the adaptive trial design characteristics; (iv) estimate the percentage of adaptive clinical trials in dialysis among all dialysis RCTs; and (v) outline temporal trends in all the above.
Methods
We performed a systematic review, reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines (PRISMA) (16). The protocol was registered with PROSPERO (CRD42020163946) and published separately (17). After testing our pre-defined search strategy (17), we found a small number (n=16) of dialysis RCTs that reported an adaptive design method. We discovered that the adaptive design methods are often not reported in the title and abstract of papers and would not be detected in a traditional systematic search. To overcome this, we developed a novel “full text systematic review” protocol and to our knowledge, this is the first use of this methodology.
Search method for the identification of trials
Electronic search – Dialysis Studies
We performed an electronic search on MEDLINE (Pubmed) from database inception until 01 June 2020. Zotero was used as our reference manager. The dialysis search terms were adapted from Beaubien-Souligny et al. 2019 (18) and included dialysis, peritoneal dialysis, hemodialysis, hemodiafiltration, haemodiafiltration, hemofiltration, haemofiltration, extracorporeal blood cleansing, haemodialysis, renal dialysis, renal replacement, end stage kidney, end stage renal, stage 5 kidney and stage 5 renal (Supplementary Table 1). The output was stored in the Research Information Systems (RIS) file format.
Machine Learning Classifier – Randomized Clinical Trials
We used the high sensitivity machine learning classifier (RobotSearch) to identify RCTs from the dialysis search output (15). RobotSearch is a machine learning classification algorithm combining an ensemble of Support Vector Machines (SVM) and Convolutional Neural Networks (CNN) with a reported Area Under the Curve of 0.987 (95% Confidence Interval (CI), 0.984 to 0.989) for RCT classification. We adjusted the parameters of RobotSearch to perform a sensitive search to increase the proportion of RCTs that are correctly identified (15). Studies classified as likely to be randomized clinical trials were sourced for the full text systematic review.
Full text Systematic Review – Adaptive Design Methods
We used Recoll for Windows to perform a full text systematic review on our dialysis randomized clinical trial search results. Recoll is based on the Xapian search engine library and provides a powerful text extraction layer and a graphical interface. The adaptive design search terms were adapted from Bothwell et al., 2018 (19) and included phase ii/iii, treatment switching, biomarker adaptive, biomarker adaptive design, biomarker adjusted, adaptive hypothesis, adaptive dose- finding, pick-the-winner, drop-the-loser, sample size re-estimation, re-estimations, adaptive randomization, group sequential, adaptive seamless, adaptive design, interim monitoring, bayesian adaptive, flexible design, adaptive trial, play-the-winner, adaptive method, adaptive AND dose AND adjusting, response adaptive, adaptive allocation, adaptive signature design, treatment adaptive, covariate adaptive and sample size adjustment (Supplementary Table 2).
Manual full text review
We then performed manual full text review to confirm studies that were included in the final systematic review. This process is summarised in a PRISMA flowchart (Figure 1). Full text review was performed by CJ, RM and CR. Disagreements were resolved by consensus and where a resolution was not reached by discussion, a consensus was reached through a third reviewer (MOD).
Inclusion/exclusion criteria for the selection of studies
Type of study design and participants
Randomized clinical trials of interventions in patients with ESKD, acute kidney injury undergoing renal replacement therapy (RRT) including haemodialysis, peritoneal dialysis, haemodiafiltration and haemofiltration. We did not limit our population to any specific disease. Additionally, we included studies that included dialysis as either a primary or secondary outcome.
Type of intervention and outcome
We did not place a restriction on the intervention type and included trials that studied medications during dialysis, medical devices, dialysis parameters and dialysis modality. We included all outcomes including surrogate markers, patient-centred outcomes and hard clinical outcomes.
Selection and analysis of trials
CJ, RM and CR extracted the study characteristics independently and in parallel. Data collected included the type of the adaptive design, stopping rule, impact of adaptive design (i.e., stopping for futility or efficacy, sample size changes, etc.), trial population, intervention, dialysis modality, the country of the lead investigator and the funder of the study (Adapted from Hatfield et al., 2016 (20))(Supplementary Table 3).
Assessment of the quality of the studies: risk of bias
We used the Cochrane Risk of Bias 2 Tool (21) to assess methodological quality of eligible trials including random sequence generation, allocation concealment, blinding of participants and health care personnel, blinded outcome assessment, completeness of outcome data, evidence of selective reporting and other biases. Risk of bias assessments were performed independently by (CJ, RM, CR, SC), and disagreements were resolved by consensus. If one or more domains was rated as high, the study was considered at high risk of bias. We summarised our findings in a risk of bias table using the Revised Cochrane risk-of-bias tool for randomized trials (22)(Supplementary Table 4).
Data synthesis
A descriptive synthesis of the data was performed. We reported overall outcomes and outcomes by (i) frequency and type of adaptive design; (ii) adaptive designs as a proportion of studies classified as dialysis RCTs by RobotSearch (iii) Population, Intervention and Outcome including dialysis modality (haemodialysis, peritoneal dialysis, haemodiafiltration and haemofiltration); (iv) publication in high impact journals; (v) geographic location and funding; (vi) reporting of adaptive design methods in title and abstract; and (vii) a risk of bias assessment.
Results
The systematic search of articles with dialysis keywords published before 01 June 2020, identified 209,033 results. 5,452 articles were classified as probable RCTs by the machine learning classifier RobotSearch (15). Full text articles were sourced (n=5,022) and we performed a full text systematic review using adaptive design keywords which identified 358 studies for manual screening. 50 studies, available as 66 articles, were included after full text review (Figure 1). 31 studies were conducted in dialysis populations and 19 studies included RRT as a primary or secondary outcome.
Study characteristics
Frequency and type of Adaptive Design
Figure 2 reports the number of adaptive design designs by year and alongside the proportion of all dialysis RCTs that used adaptive design methods. The absolute amount of dialysis trials using adaptive designs has increased each year, but this has not matched the overall increase in dialysis trials and resulted in a relative decrease over time in the use of adaptive design methods in dialysis trials ranging from 6.1% in 2009 to 0.3% in 2019 with a mean of 1.76%.
Group sequential designs were the most common type of adaptive design method used, 31 (62%) trials (22 (71%) in dialysis populations and 9 (47.4%) in dialysis outcome trials) (Table 1). The O’Brien- Fleming stopping boundary was the most common stopping rule, used in 8 trials (25.8%), followed by Lan DeMets used in 7 trials (22.6%). 18 trials (58.1%) were impacted by the use of group sequential adaptive design including 6 trials (33.3%) that stopped early for futility, 3 trials (16.7%) that stopped early for efficacy and 4 trials (22.2%) that stopped early for safety. Sample-Size Re- estimation was the second most common type of adaptive design, used in 14 trials (28%) (8 (25.8%) in dialysis populations and 6 (31.6%) in dialysis outcome trials) (Table 2). 8 trials (57.1%) were impacted by the use of Sample-Size Re-estimation adaptive design including 5 trials (62.5%) that increased sample size. Phase II/III seamless design was the third most common type of adaptive design, 3 trials (6%) (1 (3.23%) in dialysis populations and 2 (10.52%) in dialysis outcome trials) (Table 3). Adaptive Dose-Escalation and Interim analysis were used in one trial each.
Population, Intervention, and Outcome studied
Acute Kidney Injury (AKI) was studied in 27 trials (54%), End Stage Kidney Disease (ESKD) was studied in 22 trials (44%) and Chronic Kidney Disease (CKD) was studied in 1 trial (2%). Figure 3 reports the number of each population under study per year and shows a larger increase in adaptive design methods in AKI populations compared to ESKD populations. Medications were the most common intervention type, evaluated in 28 trials (56%), followed by Dialysis Modality in 6 trials (12%), and Dialysis Parameter in 4 trials (8%). Haemodialysis was the most common dialysis modality studied in 26 trials (52%), followed by Haemodialysis and haemodiafiltration in 8 trials (16%), Haemodialysis, haemodiafiltration and haemofiltration in 7 trials (14%) and Peritoneal Dialysis in 4 trials (8%). Hard clinical outcomes were selected in 30 trials (60%), followed by surrogate outcomes in 17 trials (34%) and mixed in 3 trials (6%). The outcome measure was continuous in 11 trials (22%) and dichotomous in 39 trials (78%). Phase III studies were the most common study phase, studied in 40 trials (80%) (Table 1-3).
Publication in High Impact Journals
30 studies (60%) were published in a high impact journal (Impact Factor > 9). 13 studies (26%) were published in the New England Journal of Medicine (NEJM), 5 studies (10%) were published in the Journal of the American Medical Association (JAMA), 4 studies (8%) were published in Trials, and 2 studies (4%) were published in the Journal of the American Society of Nephrology (JASN).
Geographic location and Funding
The most common country of the lead author was the United States of America in 20 studies (40%), followed by Germany in 6 studies (12%), France in 4 studies (8%), The Netherlands in 4 studies (8%), Australia in 3 studies (6%), and the United Kingdom in 3 studies (6%) (Tables 1-3). 43 studies (86%) were multicentre trials. 26 studies (52%) were supported by public funding, 17 studies (34%) were supported by private funding, 5 studies (10%) were supported by both public and private funding and 2 studies (4%) did not report the source of funding.
Reporting of Adaptive Design Method in Title and Abstract
41 studies (82%) did not report their adaptive design method in the title or abstract and would not be detected by a standard systematic review search.
Risk of Bias
Risk of bias was assessed for forty trials (protocols were excluded) (Supplementary Figure 1, Supplementary Table 4). Overall risk of bias was deemed to be “low” in 17 trials (42.5%), “some concerns” in 13 trials (32.5%), and “high risk” in 10 trials (25%). The randomization process led to some concerns for 10 studies (25%). Deviations from intended interventions led to some concerns for 4 studies (10%) and “high risk” for 6 studies (15%). Missing outcome data were deemed to be “some concerns” for 2 studies (5%) trials and ‘high risk’ of bias for 2 studies (5%). Measurement of outcome measures were deemed to be “some concerns” for 2 studies (5%) trials and ‘high risk’ of bias for 1 study (2.5%). Selection of the reported result were deemed to be “some concerns” for 6 studies (15%) trials and ‘high risk’ of bias for 1 study (2.5%).
Discussion
In this systematic review, we report that adaptive design methods were used in 50 dialysis randomized clinical trials over a 20-year period. While the absolute number has increased over time, the relative use of adaptive design methods in dialysis RCTs has decreased.
First, we report that the relative proportion of adaptive design methods in dialysis trials has reduced over time. The absolute amount of dialysis trials using adaptive designs has increased each year, but this has not matched the overall increase in dialysis trials and therefore resulted in a relative decrease. We were unable to compare this result with other specialities because recent systematic reviews have not reported the relative use of adaptive designs (20,23). Second, we report that group sequential designs are the most used type of adaptive design in dialysis trials. This is similar to previous systematic reviews in cardiology (24), oncology (23) and in a review of registered clinical trials covering multiple specialities on clinicaltrials.gov (20). Third, we report that adaptive designs were more common in Acute Kidney Injury (AKI) (54% of trials) than End Stage Kidney Disease (ESKD) (44% of trials). This may reflect increasing use of adaptive design methodology in critical care (25) and sepsis related trials (26), where AKI is most common. There were very few trials of CKD with a dialysis outcome (2%) that used an adaptive design. Many reasons for the paucity of CKD trials have been previously suggested including the use of treatments in CKD despite a lack of evidence, difficulty recruiting to CKD trials due to stringent eligibility criteria and underpowered subgroup analysis (3,27). The infrequent usage of adaptive designs in CKD trials may become a self- perpetuating barrier to using adaptive designs in future trials (20). Fourth, we report that adaptive design methods impacted the conduct of the randomised trial in the majority of studies (52%). For example, 18 (58.1%) trials were impacted by the use of group sequential adaptive design including 6 trials (33.3%) stopped early for futility, 3 trials (16.7%) stopped early for efficacy and 4 trial (22.2%) stopped early for safety. This finding is similar to a systematic review of published and publicly available trials where the most common reason for stopping group sequential trials was futility (19). Fifth, we found that most common country of the lead author was the United States of America, 20 studies (40%) and the most common funding source was public, 26 studies (52%). This finding was different to a systematic review of published and publicly available trials where 65% of trials reported industry funding (19). Funding for kidney research reached an all-time low in 2013 (4), but this has recently changed in the United States with advocacy from scientific societies such as the American Society of Nephrology, an executive order to reform the US End-Stage Kidney Disease treatment industry was signed in 2020 (28). Adaptive designs are one part of the solution for optimizing the design of clinical trials in dialysis and nephrology and will benefit from the improvement in the funding landscape (27).
Limitations
Our study has several limitations. First, we limited our search to one database (Pubmed) due to the scale of studies sourced (209,033 results). This was a deviation from our protocol but necessary to make this full text review feasible. Second, we decided to include randomised clinical trials with RRT outcomes in addition to patients currently on dialysis. This permitted a more comprehensive review of the full landscape of AKI, ESKD and CKD trials, but was a deviation from our original protocol. Third, the denominator for calculating the proportion of adaptive designs in all dialysis RCTs, will include some false negatives i.e., either not RCTs or nor dialysis. We modified the parameters of the machine learning classifier to perform a sensitive search to include as many true positives as possible. We expect this mis-classification bias to be independent of time and bias every year equally and therefore not affect the trend.
Strengths
We developed a novel full text systematic review search strategy. 41 studies (82%) did not report their adaptive design method in the title or abstract and would not be detected by a standard systematic review search methodology. This could introduce a pseudo reporting bias where adaptive design methods are reported in the main paper but not in the abstract. Our novel strategy combined classical systematic review, machine learning classifiers and a novel full text systematic review. This new method has broad applications in medical evidence synthesis and evidence synthesis in general.
Conclusion
Adaptive design methods improve efficiency of randomised clinical trials in dialysis, but their relative use is decreasing over time. Greater knowledge of adaptive design examples in dialysis will further improve uptake in dialysis randomised clinical trials.
Data Availability
Extracted data is available on request.
Author contributions
CJ, RM and MOD designed the study. CJ drafted the manuscript. CJ, RM, CR, SC, AS, MOH, MOD reviewed and approved the final version of the manuscript.
Disclosures
None
Funding
This work was performed within the Irish Clinical Academic Training (ICAT) Programme, supported by the Wellcome Trust and the Health Research Board (Grant Number 203930/B/16/Z), the Health Service Executive, National Doctors Training and Planning and the Health and Social Care, Research and Development Division, Northern Ireland. The funding source had no role in the study design, analysis or writing of report.
Acknowledgments
None
References
- 1.↵
- 2.↵
- 3.↵
- 4.↵
- 5.↵
- 6.↵
- 7.↵
- 8.↵
- 9.↵
- 10.↵
- 11.↵
- 12.↵
- 13.↵
- 14.↵
- 15.↵
- 16.↵
- 17.↵
- 18.↵
- 19.↵
- 20.↵
- 21.↵
- 22.↵
- 23.↵
- 24.↵
- 25.↵
- 26.↵
- 27.↵
- 28.↵
- 29.
- 30.
- 31.
- 32.
- 33.
- 34.
- 35.
- 36.
- 37.
- 38.
- 39.
- 40.
- 41.
- 42.
- 43.
- 44.
- 45.
- 46.
- 47.
- 48.
- 49.
- 50.
- 51.
- 52.
- 53.
- 54.
- 55.
- 56.
- 57.
- 58.
- 59.
- 60.
- 61.
- 62.
- 63.
- 64.
- 65.
- 66.
- 67.
- 68.
- 69.
- 70.
- 71.
- 72.
- 73.
- 74.
- 75.
- 76.
- 77.
- 78.
- 79.
- 80.
- 81.
- 82.
- 83.
- 84.
- 85.
- 86.
- 87.
- 88.
- 89.
- 90.
- 91.
- 92.
- 93.
- 94.