TY - JOUR T1 - Introducing the EMPIRE Index: A novel, value-based metric framework to measure the impact of medical publications JF - medRxiv DO - 10.1101/2021.04.13.21255419 SP - 2021.04.13.21255419 AU - Avishek Pal AU - Tom Rees Y1 - 2021/01/01 UR - http://medrxiv.org/content/early/2021/04/20/2021.04.13.21255419.abstract N2 - Article-level measures of publication impact (alternative metrics or altmetrics) can help authors and other stakeholders assess engagement with their research and the success of their communication efforts. The wide variety of altmetrics can make interpretation and comparative assessment difficult; available summary tools are either narrowly focused or do not reflect the differing values of metrics from a stakeholder perspective. We created the EMPIRE (EMpirical Publication Impact and Reach Evaluation) Index, a value-based, multi-component metric framework for medical publications. Metric weighting and grouping were informed by a statistical analysis of 2891 Phase III clinical trial publications and by a panel of stakeholders who provided value assessments. The EMPIRE Index comprises three component scores (social, scholarly, and societal impact), each incorporating related altmetrics indicating a different aspect of engagement with the publication. These are averaged to provide a total impact score and benchmarked so that a score of 100 equals the mean scores of Phase III clinical trial publications in the New England Journal of Medicine (NEJM) in 2016. Predictor metrics are defined to estimate likely long-term impact. The social impact component correlated strongly with the Altmetric Attention Score and the scholarly impact component correlated modestly with CiteScore, with the societal impact component providing unique insights. Analysis of fresh metrics collected 1 year after the initial dataset, including an independent sample, showed that scholarly and societal impact scores continued to increase, whereas social impact scores did not. Analysis of NEJM ‘notable articles’ showed that observational studies had the highest total impact and component scores, except for societal impact, for which surgical studies had the highest score. The EMPIRE Index provides a richer assessment of publication value than standalone traditional and alternative metrics and may enable medical researchers to assess the impact of publications easily and to understand what characterizes impactful research.Competing Interest StatementTomas Rees is an employee of Oxford PharmaGenesis, which received funding from Novartis Pharma AG for this work. Avishek Pal is an employee of Novartis Pharma AG.Clinical TrialThis is an analysis of metrics data on clinical trial publications, not a clinical trial.Funding StatementResearch and editorial support (manuscript proofreading, figure drawing, and project management) was funded by Novartis Pharma AG (https://www.novartis.com) and conducted by Oxford PharmaGenesis, who jointly planned the study, collected and analysed the data, made the decision to publish and prepared the manuscript.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:Not applicable.All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesAll data used in the analysis, as well as data points for the figures, are available on Figshare https://figshare.com/articles/dataset/EMPIRE_Index_development_data/14256305 ER -