Trial participants are frequently excluded based on their symptoms rather than their condition: A systematic review of Cochrane reviews and their component trials

1. Introduction

The principal role of a randomised controlled trial (RCT) is to evaluate whether a medical intervention is safe and effective. In order for this to happen, it is imperative that researchers measure outcomes which are both appropriate and relevant to the population of interest. Although randomised controlled trials remain the gold standard tool for treatment evaluation, many, through poor design, contribute to the overwhelming problem of waste in research [1-4]. In the Lancet collection of papers on waste in medical research, it was estimated that $240 billion of annual research expenditure is wasted [3, 5-8]. It is indeed true that much work is being done to reduce this figure, however, there is still much room for improvement [9, 10]. Inefficient studies that fail to address questions that matter to both patients and stakeholders emphasise the importance that we need to do less, but better, research [11].

Often, in the case of gynaecological conditions such as Polycystic Ovarian Syndrome (PCOS) and Endometriosis, patients require different things from their care, at different stages of their lifetime [12-15]. Not all patients with the same diagnosis will experience all of the associated complications, and, although their most bothersome symptom might differ, it is common to receive the same treatment. If we take PCOS, for example, the Rotterdam criteria are the most widely used classification for diagnosis and proposes that PCOS is present if the patient has at least two of the three characteristics: oligo- and/or anovulation, clinical and/or biochemical hyperandrogenism, and polycystic ovaries on ultrasound [16]. These criteria for the diagnosis of PCOS in itself has consequences, as by definition, not all patients with PCOS have all the possible manifestations of the disorder and therefore do not experience the same symptoms and health risk factors [12]. For endometriosis, patients may have symptoms dominated by pelvic pain, infertility, or both. In the post- reproductive era the reduction in quality-of-life from menstrual dysfunction may predominate. In the likely scenario that potential trial participants have no, or little, symptoms in common, it presents the problem of how researchers select relevant, patient-important outcomes and design RCTs that are not wasteful.

One of the most notable inclusions to the movement to reduce waste in research is the development of core outcome sets (COS). The Core Outcome Measures in Effectiveness Trials (COMET) Initiative encourages the application of agreed standardised sets of outcomes. These outcome sets represent the minimum that should be measured and reported in clinical trials in specific clinical areas [17-19].

Recently developed core outcome sets in both Polycystic Ovarian Syndrome (PCOS) and endometriosis undoubtedly violate the concept of COS, as authors in both instances concluded that not all outcomes could be reported in all trials [20, 21]. This is demonstrative of the fact that in the field of gynaecology, it is not one-size-fits-all.

This prompts the question of how we should design trials so as to incorporate this heterogeneity, as well as the implications for systematic reviews and meta-analysis. Due to the multifactorial nature of a patient’s symptoms, the design and recruitment of gynaecological trials is challenging and is approached in different ways. We conducted a systematic review to investigate how diverse symptoms and patient populations are currently handled in gynaecological trials, in which the intervention could plausibly be used to treat all symptoms related to the diagnosis. We aimed to identify the methodological strategies applied within randomised controlled trials in Cochrane Reviews, where an intervention is hypothesised to have potential benefit for patients with the index condition.

2. Methods

The study design was a systematic review with descriptive statistics. Our overall approach was to identify systematic reviews in the conditions of interest and to examine the characteristics and methodological practice of their included and excluded trials. Protocol registration was with the Prospective Register of Systematic Reviews (registration number: CRD42022334776). Full details of methods are given there but summarised below.

In February 2022 searches were undertaken to identify systematic reviews contained in the Cochrane Library on interventions for PCOS or endometriosis. We considered trials list under both ‘included’ and ‘excluded’ categories.

2.2 Data extraction

Two reviewers (KS and AW) screened all titles and abstracts against the inclusion criteria. Any disagreements were resolved through discussion.

Outcomes categories were pre-specified (appendix), along with whether each outcome was primary, secondary, or unspecified. This information was also extracted for each of the trials, along with the setting, intervention type, funding source, design, size (number of participants randomised), number of participants excluded for symptom/outcome-related reasons and the number of participants contributing to the primary outcome.

Seven different strategies were anticipated for the RCTs, as demonstrated in Figure 1. We developed these strategies as part of an iterative process. As a pilot, we developed some potential strategies we had seen during our time as researchers, assessed several trials and determined if the strategy used by each trial was different to those we anticipated. We then discussed and re-evaluated until no new strategies were found. We note that this may not be an exhaustive list, and we considered whether any additional strategies not included in our list were used. Each trial was categorised according to the strategy used, in the order listed, i.e. if the trial did not fit the criteria to be the first strategy, it was considered for the next, and so on.

• Download figure
• Open in new tab

Figure 1.

Strategies for randomised controlled trials in gynaecology.

For ease of context and reference, we named the strategies and will refer to them as such throughout:

• Participant-specific Outcome Strategy - patient-specific primary outcome was chosen, for example patient satisfaction or success.

• Composite Outcome Strategy - composite outcome, for example the resumption of menstruation or hirsutism improvement.

• Universal Eligibility Strategy - measured outcomes regardless of their relevance to the patients in the study, e.g. a study which did not restrict to patients experiencing amenorrhea (a lack of menstruation) but measured resumption of menstrual period as an outcome in every patient.

• Grouped Universal Eligibility Strategy would be trials designed the same way as the Universal Eligibility Strategy, but the statistical analysis would be restricted to those who experienced amenorrhea at baseline, as a subset analysis.

• Restricted Eligibility Strategy was allocated where a trial measured only clinical outcomes based on the symptoms of all patients in the study, i.e. a fertility trial, interested only in fertility-based outcomes, where the inclusion criteria is patients experiencing subfertility.

• Downstream Eligibility strategy was studies measuring only quality-of-life outcomes.

• Upstream Eligibility strategy was studies reporting no clinical outcomes, only biomarker outcomes.

For each trial, where available, we recorded the number of potentially eligible participants excluded as a direct result of the chosen strategy relative to the achieved sample size. That is, how much larger could the recruitment have been without an outcome-defined restriction on eligibility criteria. Where available, the number of potentially eligible participants were taken from the CONSORT flow chart. This was taken to be the number of participants that were found to be ineligible for outcome-related reasons pre-randomisation. Similarly, for each Cochrane review we recorded the numbers of identified RCTs excluded from consideration for reasons such as not reporting review-specific outcomes. To determine which trials were excluded from each Cochrane review and the reason for exclusion, we used the ‘Characteristics of excluded studies’ section of the review. We selected all excluded trials for assessment where the reason for exclusion was related to the population of patients, outcome reporting or selection.m

3. Results

4. Discussion

We conducted our systematic review to examine the current strategies used for the design of and recruitment to randomised controlled trials, by researchers. We focused on PCOS and endometriosis as exemplar conditions due to the heterogeneity of their symptoms, however, we anticipate the findings to be applicable to gynaecological research in general, and in other conditions where patients experience varying symptoms.

Systematic reviews are considered the gold standard of evidence for decision-making, used to collate, critique and summarise evidence. However, we found that the Cochrane reviews included in this review, excluded many trials, based on the outcomes they reported. Whilst at first glance, choosing a research question, and selecting trials that report outcomes relating to that research question does not appear to be erroneous, there are issues with this. Our review considered only Cochrane reviews of interventions which could, plausibly, be used regardless of symptoms. Where authors exclude trials based on their outcomes, they are reducing the number of patients with data available to add to the evidence base. Dwan and colleagues have previously discussed the prevalence and impact of excluding studies from reviews on the basis of the relevance of outcome data, with further works observing that doing so leads to potentially biasing the conclusions of systematic reviews, along with waste in production and reporting of research [23, 24]. Therefore, it would be advantageous for reviews to consider trials regardless of reported outcomes, and instead consider a multitude of symptoms and outcomes that could plausibly be treated with the intervention under investigation. This could in turn reduce the number of reviews needed, and hence waste from duplication of effort.

In order to tackle the demonstrated waste in gynaecological trials we need to focus first on improving the quality of the research questions, ensuring they matter to patients and clinicians. Three of the strategies we anticipated: Participant-specific Outcome, Composite Outcome, and Downstream Outcome were not being utilised and require consideration. It is likely that these strategies are not currently perceived as attractive to use as those that are more common. Participant-specific outcomes, such as measuring a patient’s most bothersome symptom or allowing a patient to set their own personalised goals, are relatively novel ideas, and although they have previously been used in gynaecological trials, further consideration is needed for their statistical advantages [25, 26].

Composite outcomes allow research to address more than one aspect of a patient’s health status, but their use is widely debated, with interpretation difficult [27-30]. Similarly, although a well-established instrument for providing evidence of an individual’s physical, emotional and social health, interpretation of treatment effect can also be difficult when using a quality-of-life measurement. Cox et al suggests that “in practice the main difficulty is likely to be in separating the real treatment-by-individual interaction from noise” [31].

In the current research landscape, we are typically seeing two main strategies employed in gynaecological trials, which we refer to as Universal Eligibility and Restricted Eligibility. In the first, outcomes measured are of relevance to everybody in the trial. This means patients are excluded due to stringent inclusion criteria and are unable to be involved in a trial of treatment that may be of clinical benefit to them. Secondly and conversely, some trials include participants which are not experiencing the symptom at baseline, resulting in the inability to provide useful information, as they cannot contribute to the outcome of interest. For example, researchers record BMI as an outcome for patients not overweight at baseline. In April 2022, the Royal College of Obstetricians and Gynaecology (RCOG) reported that gynaecological care waiting lists in England had grown the most substantially compared to any other medical specialty, seeing an over 60% increase on pre-pandemic levels [32].

There is no shortage of people seeking and requiring gynaecological treatment. Both of the strategies this review identified as most prevalent in gynaecological research do not utilise the potential population and are inefficient, often leading to research that lacks in impact and results in waste.

5. Conclusions

As a minimum, participants whose symptoms could potentially benefit from any specific treatment should have a chance to receive said treatment. This research identified multiple sources of waste in the current gynaecological research landscape. We have shown that the population of patients available is often under-utilised by excluding patients based on the outcomes measured, or alternatively, researchers are measuring outcomes in patients who do not experience the associated symptom(s).

Gynaecological patients experience heterogeneity in their symptoms and therefore it is crucial to employ patient-specific outcome measures. Not only would this reduce research waste from a budget and resource perspective, it would also, most importantly, alleviate patient burden.

Authors’ contributions

Katie Stocking: Conceptualization; Data curation; Formal analysis; Methodology; Writing – Original draft preparation. Andrew Watson: Conceptualization; Data curation; Validation; Writing – Reviewing and editing. Jamie J Kirkham: Conceptualization; Methodology; Supervision; Writing – Reviewing and editing. Jack Wilkinson: Methodology; Supervision; Writing – Reviewing and editing Andy Vail: Conceptualization; Methodology; Supervision; Writing – Reviewing and editing.

Funding

Katie Stocking, Doctoral Research Fellow (NIHR301756), is funded by the National Institute for Health and Care Research (NIHR) for this research project. The views expressed in this publication are those of the author(s) and not necessarily those of the NIHR, NHS or the UK Department of Health and Social Care.