Abstract
Background Peer review processes may inadequately assess compliance with established reporting guidelines such as the Consolidated Standards of Reporting Trials (CONSORT) criteria. Large language models (LLMs) demonstrate potential for systematic manuscript evaluation; however, their accuracy in detecting adherence to CONSORT guidelines in published clinical trials remains unexplored.
Methods This cross-sectional study evaluated the compliance of 20 randomized controlled trials published between 2015 and 2024 from immunology journals, identified through PubMed, with the CONSORT 2010 guidelines. Three large language models (ChatGPT-4o, Gemini 2.5 Pro, and Claude Sonnet 4) independently assessed compliance across 37 CONSORT subpoints. The primary endpoint was the mean CONSORT compliance percentage. Secondary endpoints included the proportion of articles meeting a 90% compliance threshold and agreement between LLM assessments. Statistical analysis employed repeated measures ANOVA with post-hoc pairwise comparisons (α = 0.05).
Results Mean CONSORT compliance rates were: ChatGPT-4o 81% (95% CI: 77-85%), Claude Sonnet 4 68% (95% CI: 61-75%), and Gemini 2.5 Pro 55% (95% CI: 48-62%). Overall compliance across all LLMs was 68% (95% CI: 64-72%). Using a 90% compliance threshold as a quality benchmark, ChatGPT-4o identified 25% of papers (5/20), Claude Sonnet 4 identified 5% (1/20), and Gemini 2.5 Pro identified none (0/20) as meeting this standard. Repeated-measures ANOVA demonstrated significant differences in LLM performance (F2,38 = 40.79, p < 0.001, partial η2 = 0.682). All pairwise comparisons between models were statistically significant (p ≤ 0.002).
Conclusions Large language models detected CONSORT compliance deficiencies in published randomized trials, aligning with previously reported rates of 60-70%, which validates their accuracy in identifying persistent reporting quality issues. The substantial variation between LLM assessments indicates the need for standardized evaluation protocols. These findings support the potential utility of LLM-assisted manuscript evaluation to improve adherence to established reporting guidelines.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This study did not receive any funding
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Data Availability Statement
Raw data supporting the findings of this study are openly available in Zenodo at https://doi.org/10.5281/zenodo.17253371
