Abstract
Importance Accurate interpretation of randomized controlled trial (RCT) results is essential for guiding clinical practice in oncology. Reporting “spin” can misrepresent treatment efficacy, potentially leading to suboptimal clinical decisions. Standardized methods could help to detect misleading reporting.
Objective To determine whether large language models (LLMs) can accurately classify oncology RCTs as positive or negative based on primary endpoint achievement when provided with different sections of the trial report, thereby assessing their utility in identifying potential spin in conclusions.
Design Methodological evaluation using LLMs and human annotations of previously published clinical trials.
Setting Random sample of RCTs from seven major medical journals published between 2005 and 2023.
Participants 250 two-arm, single primary endpoint oncology RCT reports were randomly selected from the specified journals and publication years.
Exposure(s) Human annotators independently classified trials based on primary endpoint results before LLM evaluation. Three commercial LLMs (GPT-3.5 Turbo, GPT-4o, and o1) classified trials based on four different text inputs: 1) conclusion only, 2) methods and conclusion, 3) methods, results, and conclusion, or 4) title and full abstract.
Main Outcome(s) and Measure(s) Performance of LLMs in classifying trials as positive or negative, primarily measured using the F1 score.
Results The analysis included 250 RCT reports; based on human annotation, 146 (58.4%) were positive and 104 (41.6%) were negative. o1 demonstrated the highest performance across all input conditions, achieving F1 scores of 0.932 (conclusion only), 0.960 (methods and conclusion), 0.980 (methods, results, and conclusion), and 0.970 (title and full abstract). Analysis of trials incorrectly classified as positive by the LLM when using only the conclusion revealed patterns such as absence of primary endpoint data, emphasis on secondary or subgroup findings, or unclear endpoint distinctions within the conclusion.
Conclusions and Relevance LLMs can accurately classify oncology RCT outcomes. Discrepancies between classifications based on conclusions versus more complete text indicate potential spin. This approach could serve as a valuable supplementary tool for researchers, reviewers, and editors to enhance transparency and critical appraisal of oncology trial reporting, though further validation is required, especially for trials with more complex designs.
Question Can large language models (LLMs) accurately classify oncology randomized controlled trials (RCTs) as positive or negative based on primary endpoint achievement, and can this help identify potential “spin” in trial conclusions?
Findings In this methodological evaluation of 250 oncology RCTs, the o1 LLM achieved high accuracy in classifying trials based on the title and full abstract, outperforming classifications based on the conclusion alone. Discrepancies between classifications using conclusions versus more complete text often indicated patterns that could be considered as spin.
Meaning LLMs show promise as a supplementary tool to detect potential spin by identifying inconsistencies between conclusions and overall results.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
No funding was received for this project.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Footnotes
Ethics approval and consent to participate: Not applicable
Availability of data and materials: All data and code used to obtain this study’s results have been uploaded to https://github.com/windisch-paul/positive_negative. The data has also been uploaded to Dryad (Reviewer link: http://datadryad.org/share/wWY4y8CkvguMpbL9XpaasFEMRbIniEe3eqwGXdmx-Sg).
Competing interests: The authors declare no conflict of interest.
Funding: No funding was received for this project.
All authors read and approved the final manuscript.
