TY - JOUR T1 - Evaluating quality improvement at scale: development of a reporting methodology for board-level insights in a UK mental health Trust JF - medRxiv DO - 10.1101/2020.02.13.20022475 SP - 2020.02.13.20022475 AU - Kia-Chong Chua AU - Barbara Grey AU - Michael Holland AU - Claire Henderson AU - Nick Sevdalis Y1 - 2020/01/01 UR - http://medrxiv.org/content/early/2020/02/14/2020.02.13.20022475.abstract N2 - Background The question of whether quality improvement (QI) in healthcare actually improves quality requires situational awareness over a diverse range of QI projects. An organisational-level overview remains largely elusive in recent literature that looks at QI at scale. On the other hand, where organisational-level estimates were offered, the extent they could be attributed to QI has been unclear. Evaluating QI at scale entails a reporting methodology that is slightly different from prevailing guidance on research evaluation of specific projects. This is essential for generating insights for organisational-level decision making to optimise return on investment from QI.Methods A researcher-in-residence worked with a resident QI team to develop and conduct a retrospective evaluation of QI projects in a large healthcare organisation in the UK that specialises in mental health. Using a survey, we enumerated project outcomes and explored costs and benefits, as well as contextual, input and process factors that might be associated with whether QI projects led to a change in routine practice.Findings Out of 52 QI projects, 10 led to a change in routine practice. Across a diverse range of projects, time invested by staff is an opportunity cost that could serve as a common denominator for return on investment. Retrospective data accrual proved problematic for measuring benefits. Odds ratios from logistic regressions show that process factors had more apparent impact on project outcomes than contextual and input factors. Of note, if QI projects had documentation for Plan-Do-Study-Act (PDSA) cycles, the odds of effecting a change in routine practice were much higher.Conclusions Prospective monitoring is necessary for ensuring systematic data accrual. Data accrual on costs and benefits of QI projects requires iterative development and learning. The onus of developing structured reports should fall on a resident QI unit. This will enable consistent measurement and coherent iterations. Developing a reporting methodology for routine organisational-level monitoring can be an asset for improving practice, one that is pivotal for a learning healthcare system.Competing Interest StatementThe authors have declared no competing interest.Funding StatementNo external funding was received.Author DeclarationsAll relevant ethical guidelines have been followed; any necessary IRB and/or ethics committee approvals have been obtained and details of the IRB/oversight body are included in the manuscript.YesAll necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).Yes I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesData is available upon request from the corresponding author ER -