Skip to main content
medRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search

DORA-compliant measures to assess research quality and impact in biomedical institutions: review of published research, international best practice and Delphi survey

View ORCID ProfileAnna R Gagliardi, Rob HC Chen, Himani Boury, Mathieu Albert, View ORCID ProfileJames Chow, Ralph S DaCosta, Michael Hoffman, View ORCID ProfileBehrang Keshavarz, Pia Kontos, Jenny Liu, Mary Pat McAndrews, Stephanie Protze
doi: https://doi.org/10.1101/2022.06.16.22276440
Anna R Gagliardi
1Toronto General Hospital Research Institute, University Health Network, Toronto, Ontario, Canada
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Anna R Gagliardi
  • For correspondence: anna.gagliardi@uhnresearch.ca
Rob HC Chen
2UHN Research Solutions and Services, University Health Network, Toronto, Ontario, Canada
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Himani Boury
1Toronto General Hospital Research Institute, University Health Network, Toronto, Ontario, Canada
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Mathieu Albert
3The Institute for Education Research, University Health Network, Toronto, Ontario, Canada
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
James Chow
4Techna Institute, University Health Network, Toronto, Ontario, Canada
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for James Chow
Ralph S DaCosta
5Princess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Michael Hoffman
5Princess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Behrang Keshavarz
6Toronto Rehabilitation Institute (KITE), University Health Network, Toronto, Ontario, Canada
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • ORCID record for Behrang Keshavarz
Pia Kontos
6Toronto Rehabilitation Institute (KITE), University Health Network, Toronto, Ontario, Canada
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jenny Liu
7Krembil Research Institute, University Health Network, Toronto, Ontario, Canada
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Mary Pat McAndrews
7Krembil Research Institute, University Health Network, Toronto, Ontario, Canada
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Stephanie Protze
8McEwen Stem Cell Institute, University Health Network, Toronto, Ontario, Canada
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Supplementary material
  • Data/Code
  • Preview PDF
Loading

ABSTRACT

Objective The San Francisco Declaration on Research Assessment (DORA) advocates for assessing biomedical research quality and impact, yet academic organizations continue to employ traditional measures such as Journal Impact Factor. We aimed to identify and prioritize measures for assessing research quality and impact.

Methods We conducted a review of published and grey literature to identify measures of research quality and impact, which we included in an online survey. We assembled a panel of researchers and research leaders, and conducted a two-round Delphi survey to prioritize measures rated as high (rated 6 or 7 by ≥ 80% of respondents) or moderate (rated 6 or 7 by ≥ 50% of respondents) importance.

Results We identified 50 measures organized in 8 domains: relevance of the research program, challenges to research program, or productivity, team/open science, funding, innovations, publications, other dissemination, and impact. Rating of measures by 44 panelists (60%) in Round One and 24 (55%) in Round Two of a Delphi survey resulted in consensus on the high importance of 5 measures: research advances existing knowledge, research plan is innovative, an independent body of research (or fundamental role) supported by peer-reviewed research funding, research outputs relevant to discipline, and quality of the content of publications. Five measures achieved consensus on moderate importance: challenges to research productivity, potential to improve health or healthcare, team science, collaboration, and recognition by professional societies or academic bodies. There was high congruence between researchers and research leaders across disciplines.

Conclusions Our work contributes to the field by identifying 10 DORA-compliant measures of research quality and impact, a more comprehensive and explicit set of measures than prior efforts. Research is needed to identify strategies to overcome barriers of use of DORA-compliant measures, and to “de-implement” traditional measures that do not uphold DORA principles yet are still in use.

Competing Interest Statement

The authors have declared no competing interest.

Funding Statement

The author(s) received no specific funding for this work

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

The University Health Network Research Ethics Board waived the need for ethical review and approval of this work.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.

Yes

Data Availability

All relevant data are within the manuscript and its Supporting Information files

Copyright 
The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY 4.0 International license.
Back to top
PreviousNext
Posted June 17, 2022.
Download PDF

Supplementary Material

Data/Code
Email

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
DORA-compliant measures to assess research quality and impact in biomedical institutions: review of published research, international best practice and Delphi survey
(Your Name) has forwarded a page to you from medRxiv
(Your Name) thought you would like to see this page from the medRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
DORA-compliant measures to assess research quality and impact in biomedical institutions: review of published research, international best practice and Delphi survey
Anna R Gagliardi, Rob HC Chen, Himani Boury, Mathieu Albert, James Chow, Ralph S DaCosta, Michael Hoffman, Behrang Keshavarz, Pia Kontos, Jenny Liu, Mary Pat McAndrews, Stephanie Protze
medRxiv 2022.06.16.22276440; doi: https://doi.org/10.1101/2022.06.16.22276440
Digg logo Reddit logo Twitter logo Facebook logo Google logo LinkedIn logo Mendeley logo
Citation Tools
DORA-compliant measures to assess research quality and impact in biomedical institutions: review of published research, international best practice and Delphi survey
Anna R Gagliardi, Rob HC Chen, Himani Boury, Mathieu Albert, James Chow, Ralph S DaCosta, Michael Hoffman, Behrang Keshavarz, Pia Kontos, Jenny Liu, Mary Pat McAndrews, Stephanie Protze
medRxiv 2022.06.16.22276440; doi: https://doi.org/10.1101/2022.06.16.22276440

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Health Systems and Quality Improvement
Subject Areas
All Articles
  • Addiction Medicine (216)
  • Allergy and Immunology (495)
  • Anesthesia (106)
  • Cardiovascular Medicine (1101)
  • Dentistry and Oral Medicine (196)
  • Dermatology (141)
  • Emergency Medicine (274)
  • Endocrinology (including Diabetes Mellitus and Metabolic Disease) (502)
  • Epidemiology (9785)
  • Forensic Medicine (5)
  • Gastroenterology (481)
  • Genetic and Genomic Medicine (2319)
  • Geriatric Medicine (223)
  • Health Economics (463)
  • Health Informatics (1563)
  • Health Policy (737)
  • Health Systems and Quality Improvement (606)
  • Hematology (238)
  • HIV/AIDS (507)
  • Infectious Diseases (except HIV/AIDS) (11657)
  • Intensive Care and Critical Care Medicine (617)
  • Medical Education (240)
  • Medical Ethics (67)
  • Nephrology (258)
  • Neurology (2149)
  • Nursing (134)
  • Nutrition (338)
  • Obstetrics and Gynecology (427)
  • Occupational and Environmental Health (518)
  • Oncology (1183)
  • Ophthalmology (366)
  • Orthopedics (129)
  • Otolaryngology (220)
  • Pain Medicine (148)
  • Palliative Medicine (50)
  • Pathology (313)
  • Pediatrics (698)
  • Pharmacology and Therapeutics (302)
  • Primary Care Research (267)
  • Psychiatry and Clinical Psychology (2188)
  • Public and Global Health (4674)
  • Radiology and Imaging (781)
  • Rehabilitation Medicine and Physical Therapy (457)
  • Respiratory Medicine (624)
  • Rheumatology (274)
  • Sexual and Reproductive Health (226)
  • Sports Medicine (210)
  • Surgery (252)
  • Toxicology (43)
  • Transplantation (120)
  • Urology (94)