Abstract
Objectives To compare how junior clinical researchers generate data-driven hypotheses with a visual interactive analytic tool for filtering and summarizing large health data sets coded with hierarchical terminologies (VIADS) or other analytical tools routinely used by participants on the same datasets.
Methods We recruited clinical researchers from all over the United States of America and separated them into “experienced” and “inexperienced” groups using predetermined criteria. Within the groups, participants were randomly assigned to a VIADS or non-VIADS groups (control) group. We recruited two participants for the pilot study and 18 for the main study. Fifteen (out of 18) were junior clinical researchers, including seven in the control group and eight in the VIADS group. All participants used the same datasets and study scripts. Each participant conducted a remote 2-hour study session for hypothesis generation. The VIADS groups also had a 1-hour training session. The same researcher coordinated the study session. Two participants in the pilot study were one experienced and one inexperienced clinical researcher. During the session, all participants followed a think-aloud protocol to verbalize their thoughts and actions during data analysis and hypothesis generation. Follow-up surveys were administered to all participants after each study session. All screen activities and audio were recorded, transcribed, coded, and analyzed. Every ten randomly selected hypotheses were included in one Qualtrics survey for quality evaluation. Seven expert panel members rated each hypothesis on validity, significance, and feasibility.
Results Eighteen participants generated 227 hypotheses, of which 147 (65%) were valid based on our criteria. Each participant generated between one and 19 valid hypotheses during the 2-hour session. The VIADS and control groups generated a similar number of hypotheses on average. It took the VIADS group participants approximately 258 seconds to generate one valid hypothesis; for the control group— it took 379 seconds; however, the difference was not statistically significant. Furthermore, the validity and significance of the hypotheses were slightly lower in the VIADS group, though not statistically significant. The feasibility of the hypotheses was statistically significantly lower in the VIADS group than in the control group. The average quality rating of hypotheses per participant ranged from 7.04 to 10.55 (out of 15). In the follow-up surveys, VIADS users provided overwhelmingly positive feedback on VIADS, and they all agreed (100%) that VIADS offered new perspectives on the datasets.
Conclusion The role of VIADS in hypothesis generation trended favorably with respect to the assessment of hypotheses generated; however, a statistically significant difference was not reached, possibly related to sample size or the 2-hour study session being inadequate. Further characterization of hypotheses, including specifics on how they might be improved, could guide future tool development. Larger-scale studies may help to reveal more conclusive hypothesis generation mechanisms.
Highlights of the paper
Identified the scientific hypothesis generation process from other parts of scientific or medical reasoning.
Conducted a human subject study to generate data-driven hypotheses among clinical researchers, recorded the process, and analyzed the results.
Established baseline data for junior clinical researchers: the number, the quality, the validity rate, and the time needed to generate data-driven hypotheses within 2 hours.
VIADS might stimulate users’ new ways of thinking during hypothesis generation.
Competing Interest Statement
The authors have declared no competing interest.
Clinical Protocols
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9345027/
Funding Statement
The project is supported by a grant from the National Library of Medicine of the United States National Institutes of Health (R15LM012941). It is partially supported by the National Institute of General Medical Sciences of the National Institutes of Health (P20 GM121342). This work has also benefited from research training resources and the intellectual environment enabled by the NIH/NLM T15 South Carolina Biomedical Informatics and Data Science for Health Equity (SC BIDS4Health) research training program (T15LM013977).
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
Institution Review Board of Clemson University granted approval for this work (IRB2020-056) . Institution Review Board of Ohio University granted approval for this work (18-X-192).
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Data Availability
All data produced in the present study are available upon reasonable request to the authors on a case by case basis.
