ABSTRACT
Objective Develop and evaluate the effect of a Fast Healthcare Interoperability Resources (FHIR) app, Health Dart, integrating information from Indiana’s community health information exchange (HIE), the Indiana Network for Patient Care (INPC), directly with Cerner, an electronic health record (EHR)
Materials and Methods Health Dart was implemented in 14 Indiana University Health emergency departments (ED) using a stepped-wedge study design. We analyzed rates of INPC use in 286,175 ED encounters between October 1, 2019 and December 31, 2020. Logistic regression was used to model the probability of INPC use given the implementation context, such as user interface (UI) enhancements and the COVID-19 pandemic.
Results INPC use increased by 131% across all encounters (from 3.6% to 8.3%; p<0.001) after Health Dart implementation. INPC use increased by144% (from 3.6% to 8.8%; p<0.001) more than two months post-implementation. After UI enhancements, post-implementation INPC use increased 123% (from 3.5% to 7.8%) compared to 181% (from 3.6% to 10.1%; p<0.001) in post-implementation encounters that occurred before UI enhancements. During the pandemic, post-implementation INPC use increased by 135% (from 3.4% to 8.0%; p<0.001) compared to 178% (from 3.6% to 10%; p<0.001) in post-implementation encounters that occurred before the pandemic. Statistical significance was determined using 95% confidence intervals (α=0.05).
Discussion Direct integration of HIE information into an EHR substantially increased frequency of HIE use, but the effect was weakened by the UI enhancements and pandemic.
Conclusion HIE information integrated into EHRs in the form of dashboards can potentially make information retrieval more efficient and effective for clinicians.
INTRODUCTION
Providing clinicians with the information they need when and in the format they need it is a significant research challenge in clinical informatics.1–4 Information needed to care for patients is typically still fragmented across too many different systems and not easy to collate, organize and review.1,5,6 This problem is aggravated in the context of increasing interoperability. Not only are clinicians tasked with thoroughly reviewing patient cases in their own electronic health record (EHR) - they are expected to do the same for patient records generated outside of their organization available through health information exchange (HIE).
HIE provides many benefits for healthcare processes and outcomes, which is why it is strongly supported by the Office of the National Coordinator and multiple other stakeholders.7–14 Among these benefits are fewer duplicated procedures, reduced imaging, lower costs, and improved patient safety.7,15 Despite these benefits, the majority of hospitals do not engage in meaningful integration of shared health data in the EHR beyond the continuity of care record.16,17
Clinicians ideally need all relevant information about a specific patient in one place.18 Missing (or inaccessible) information can have detrimental consequences for care.19 Community health information exchanges typically provide broad, comprehensive coverage of patient information from regional healthcare organizations, regardless of which EHR they use.20
Vendor-mediated HIEs, on the other hand, such as Care Everywhere in Epic and Commonwell in Cerner, and collaboratives such as Carequality, are often more constrained in their scale and scope of information coverage – making it sometimes difficult or impossible for clinicians to access all relevant information about a patient.20
Clinicians typically access information in the HIE through a separate application or portal because most HIEs are not directly integrated with EHRs. However, the resulting multiple logins, workflow interruptions, and poor presentation often impede effective and efficient information retrieval, resulting in a low level of use, especially in emergency medicine.21–25 Vendor-mediated HIEs often preferentially integrate external information in the EHR only when it comes from their customers, leaving information from other EHRs more difficult to access.20 Early evidence shows that integrating HIE information directly into the EHR can substantially increase the use of information from the HIE.25,26
Retrieving patient information through most HIE portals is subject to an additional limitation: Clinicians must typically browse through information organized by type, such as medications, labs, orders or physician notes, and time, because like EHRs, HIEs do not offer problem-oriented views.27 Problem-oriented views have been shown to improve data retrieval workflows, allowing providers to complete EHR tasks more efficiently, with fewer errors and cognitive task load, and greater user satisfaction.28–31
These limitations have two major consequences. First, the lack of widescale interoperability and barriers to information access imposes an exhausting litany of clerical tasks on clinicians which contributes to burnout and waste.32,33 Second, clinicians routinely forgo searching for and retrieving additional clinical data about patients,34 which contributes to waste and adverse patient outcomes.
The goal of this study, therefore, was two-fold: (1) develop a Fast Healthcare Interoperability Resources (FHIR) app, Health Dart, that integrates information from Indiana’s community HIE, the Indiana Network for Patient Care (INPC), directly with an EHR in the form of a problem-oriented dashboard; (2) determine how the implementation of Health Dart affected INPC use among clinicians at 14 Indiana University (IU) Health emergency departments (EDs) using a stepped-wedge trial design, allowing us to causally relate Health Dart with changes in INPC use.
MATERIALS AND METHODS
Health Dart development and pilot implementation
We developed the Health Dart application (app) (Figure 1) based on an earlier version focused on chest pain.25 Health Dart uses the FHIR standard to integrate highly relevant information from the INPC directly into Cerner, the EHR used at Indiana University (IU) Health and its EDs. Prior to the implementation of Health Dart, IU Health clinicians’ only means of accessing the INPC was via a web-based tool called CareWeb.35 After navigating to CareWeb from Cerner*, clinicians had to browse or search through the patient’s health records to find relevant information. Health Dart, on the other hand, retrieves the information from the INPC that is most relevant to seven chief complaints (chest pain, abdominal pain, weakness/dizziness/headache, back/flank pain, pregnancy, arrhythmia, and dyspnea) and integrates it into Cerner in the form of a chief complaint-focused dashboard. A group of ED physicians, led by JS, identified the information to be displayed for each chief complaint. The supported chief complaints are the reason for approximately 40% of all encounters across the IU Health ED system and common in other EDs.36
Health Dart screenshot displaying patient data for the chief complaint of chest pain
Clinicians access Health Dart directly within Cerner PowerChart with one click, allowing them to launch the app without navigating to CareWeb, and possibly having to re-enter user credentials and search for/select a patient. Health Dart is an additional way to access information in the INPC. The traditional access to CareWeb is still available if clinicians need to review the full patient record in the INPC application. Health Dart was implemented at a central pilot site, IU Health’s Methodist ED, in January 2018 before its launch across the whole IU Health system.
Study Design and Setting
After pilot implementation, Health Dart was implemented in 14 IU Health EDs with 166 clinicians using a stepped-wedge, cluster non-randomized controlled study design. As an alternative to randomized controlled trials, which are often not practical for site-based studies of healthcare delivery interventions, this scientifically validated, pragmatic study design enables causal inference from the multi-site intervention rather than merely establishing an association.37–39 Although the app was available within Cerner across all EDs beginning in April 2018, formal implementation, including structured advertisement via email and in-person or virtual training on how to use the app, did not begin until December 2019. RR, assisted by HH, led the formal implementation in partnership with the IU Health Chief Medical Information Officer (JS) and ED site directors. Implementation occurred in four waves, with each wave lasting two-months in duration and comprising three to four ED sites located in the same geographic region. Each ED served first as a control and subsequently as a test site, enabling causal inference about the effects of Health Dart on INPC use. Health Dart was implemented in four Wave 1 sites on December 13, 2019; three Wave 2 sites on February 7, 2020; three Wave 3 sites on June 8, 2020; and four Wave 4 sites on August 3, 2020. Implementation was paused for two months between Waves 2 and 3 due to the COVID-19 pandemic.
Conceptual Models
We used the Unified Theory of Acceptance and Use of Technology (UTAUT) and Consolidated Framework for Implementation Research (CFIR) as our conceptual models. They identify implementation context (the setting and circumstances of the implementation) as one of the predominant indicators of user behaviors, adoption and sustainable use of HIE.40,41,16 In our analysis of the effect of the implementation of Health Dart on INPC use we focused on implementation context, which included enhancements to the UI, repositioning of the app in the toolbar, and the COVID-19 pandemic.
Measures
We evaluated INPC use for the 286,175 ED encounters that occurred between October 1, 2019, and December 31, 2020, at all sites collectively. User log data and encounter details were extracted from the INPC and IU Health’s Enterprise Data Warehouse by the Regenstrief Institute Data Core. Encounters were categorized by implementation wave (Wave 1-4), INPC use (yes or no), and INPC access method (CareWeb or Health Dart). Encounters were also categorized as occurring pre-or post-implementation, before or after UI enhancements were made to the app (May 11, 2020), before or after the app was moved to a more prominent position on the toolbar (October 14, 2020), and before or during the COVID-19 pandemic as defined by the date of the first reported COVID-19 case in Indiana (March 6, 2020). To understand the temporal effects of app implementation, encounters were also grouped by three different time periods relative to implementation: pre-implementation, between 0 and 2 months post-implementation, and more than 2 months post-implementation. The outcome was a binary variable representing whether the INPC (either through Health Dart or CareWeb) was used in an encounter.
Statistical Analysis
First, a descriptive analysis provided rates of INPC use at each ED site and implementation wave over the study period (October 1, 2019, to December 31, 2020). INPC use rates at each of the ED sites were calculated and plotted for each month of the study period to visualize the effects of Health Dart and implementation context (UI enhancement, toolbar repositioning, and COVID-19 pandemic) on INPC use. For each site, we also determined the proportion of INPC use that occurred through Health Dart versus CareWeb.
Second, we determined the associations between INPC use and variables describing the implementation context. Using a chi-square test, we compared INPC use rates between pre- and post-Health Dart implementation groups for all encounters and stratified them by the implementation wave, UI enhancements, toolbar repositioning, and COVID-19 pandemic. Chi-square and Cochran-Armitage Trend tests were used to compare and assess changes in INPC use rates across the three time periods (pre, 0-2 months, and >2 months) for all encounters, stratified by implementation wave, UI enhancements, and COVID-19 pandemic. A chi-square test was used to assess whether increases in INPC use were associated with the repositioning of the app on the toolbar. Because this change was made after Health Dart was implemented in all four waves, we did not compare INPC use rates among the three time periods while adjusting for the toolbar repositioning and could not separate the effect of the app implementation from the toolbar repositioning.
Third, logistic regression modeling was used to estimate the effect of Health Dart implementation on the probability of INPC use while adjusting for implementation wave, UI enhancements, and pandemic status. Specifically, we considered the following four models with different combinations of independent variables: Model A) the binary Health Dart implementation variable and implementation wave; Model B) variables in Model A, the binary variables of the UI enhancement, and pandemic status and their interactions with the Health Dart implementation variable; Model C) the three-level categorical variable of different Health Dart implementation time periods and implementation waves; and Model D) variables in Model C, the binary variables of the UI enhancement and pandemic status, and their interactions with the three-level categorical variable of different Health Dart implementation time periods. In all models, a site-level random intercept was included to account for clustering effects within the same sites.
A power analysis was conducted based on the stepped-wedge cluster non-randomized design and pilot study results to test the hypothesis that app implementation increased INPC use.25 A study site having at least 39 daily encounters per two-month implementation wave (a total of 2,340 encounters) would result in the conclusion that Health Dart led to greater INPC usage under a stringent test with a significance level of α ≤ 0.01 powered at 0.9. Given the large encounter volume at IU Health EDs (286,175 ED encounters) during the study period, there was adequate power to detect a statistically significant difference in INPC use due to the app implementation.
The implementation context is an important predictor of user behavior in the UTAUT and CFIR models, and we therefore modeled its influence on INPC use. The implementation context included introducing enhancements to the user interface (UI), repositioning the app to a more prominent location on the EHR toolbar, and the COVID-19 pandemic. The COVID-19 pandemic is an especially notable event as the National Syndromic Surveillance Program reported a 42% decline in ED visits between March 29 and April 25, 2020, compared to the same time period in 2019.42 During this time, many common ED chief complaints were displaced by infectious diseases and respiratory conditions compared to the prior year.42,43 In our study, we anticipated an incremental increase in overall INPC use at the beginning of the Health Dart implementation and a leveling off over time. We predicted a similar INPC use pattern following the UI enhancements and the repositioning of the app in the toolbar. Because there was a reduction in opportunities to use Health Dart for its intended purpose due to COVID-19, we expected an initial decrease in INPC use shortly after the start of the pandemic with a gradual increase and leveling off over time.
This study was reviewed and approved by the Indiana University Institutional Review Board (protocol #1905749709). Analyses were completed using SAS 9.4 (SAS Institute Inc., Cary, NC, USA). Statistical significance was determined using 95% confidence intervals (alpha level=0.05).
RESULTS
INPC Use by Implementation Wave
The average rate of INPC use at the 14 sites throughout the study period (October 1, 2019 to December 31, 2020) ranged from 1.2% in Wave 3 to 18.1% in Wave 1. Rates were similar among EDs in the same implementation wave except for IU Health Morgan in Wave 4, which used INPC at a higher rate than the other Wave 4 sites. In all waves, INPC use initially increased after implementation and continued to increase after UI enhancements and toolbar repositioning. The odds of INPC use increased 10% after the toolbar changes (p<0.001). In Wave 1, there was a decline in use of the INPC in March 2020 coinciding with the start of the COVID-19 pandemic with a gradual return to pre-pandemic rates over time; the effect of the pandemic was not as dramatic at the other sites (Figure 2). Across all four implementation waves, there was a pattern of incremental increases in INPC use in the 0 to 2 months and >2 months post-implementation time periods. At four sites (Tipton, Blackford, Jay, and Paoli), INPC use leveled off between the 0 to 2 months and >2 months post-implementation time periods (Figure 3).
Monthly rates of HIE use in each Health Dart implementation wave. Health Dart was implemented in four Wave 1 sites on December 13, 2019; three Wave 2 sites on February 7, 2020; three Wave 3 sites on June 8, 2020; and four Wave 4 sites on August 3, 2020. The implementation study was temporarily paused between Wave 2 and Wave 3 due to the COVID-19 pandemic. 110 encounters had missing data (0.04%). The Health Dart application was available in Cerner in April 2018; therefore, we could not prevent use of the application prior to the study period for each wave. Abbreviations: UI, user interface; HIE, health information exchange.
Rate of HIE use before and after Health Dart implementation at 14 IUH EDs in 4 implementation waves. HIE use combined access through the web-based portal and Health Dart.
INPC Use Before and After Health Dart Implementation
Clinicians’ INPC use was 131% higher overall (from 3.6% to 8.3%; p<0.001) in post-implementation encounters compared to pre-implementation encounters (Table 1). The increased likelihood of INPC use in post-implementation encounters was also observed in each implementation wave, in pre- and post-UI enhancement periods, and before and during the COVID-19 pandemic. The effect of the implementation was not as strong in encounters that occurred after UI enhancements or in encounters that occurred during the pandemic. After UI enhancements, post-implementation INPC use increased 123% (from 3.5% to 7.8%; p<0.001) compared to 181% (from 3.6% to 10.1%; p<0.001) in post-implementation encounters that occurred before UI enhancements. During the pandemic, post-implementation INPC use increased by 135% (from 3.4% to 8.0%; p<0.001) compared to 178% (from 3.6% to 10%; p<0.001) in post-implementation encounters that occurred before the pandemic.
Comparison of INPC use pre- and post-Health Dart implementation
As was observed in the descriptive analysis, an INPC use rates over the pre-implementation, 0 to 2 months post-implementation, and >2 months post-implementation time periods increased across all encounters (p<0.001) (Table 2). This trend was also observed in each implementation wave, in pre- and post-UI enhancement periods, and before and during the COVID-19 pandemic. In each of the time periods, the effect of Health Dart on INPC use was not as strong in encounters that occurred after UI enhancements (p<0.001) or in encounters that occurred during COVID-19.
Comparison of INPC use stratified by pre-, 0 to 2 months post, and more than 2 months post-Health Dart implementation time periods
Estimating the Probability of INPC Use
The odds of INPC use increased 68% post-Health Dart implementation, adjusted for the implementation wave (p<0.001) (Table 3). The odds of INPC use was 47% higher at 0 to 2 months post-implementation (p<0.001) and 76% higher at >2 months post-implementation (p<0.001) compared to pre-Health Dart, after controlling for wave.
Modeling the probability of INPC use
The odds of INPC use was 38% higher after the Health Dart implementation and before UI enhancements (p<0.001). The effect of Health Dart strengthened after UI enhancement, with the odds increasing to 47% (p<0.001). There was an increasing trend of odds of INPC use over time post-Health Dart implementation both before and after UI enhancement. Before the UI enhancement, the odds of INPC use was 26% higher at 0 to 2 months post-implementation (p<0.001) and 37% higher at >2 months post-implementation (p<0.001) compared to pre-Health Dart. After the UI enhancement, the odds of INPC use was 58% higher at 0 to 2 months post-implementation and 40% higher at >2 months post-implementation (all p-values<0.001).
Before the COVID-19 pandemic, the odds of INPC use was 46% higher after Health Dart implementation (p<0.001). The effect of Health Dart slightly weakened during the COVID-19 pandemic, when the odds of INPC use was 39% higher after Health Dart implementation (p<0.001). Before the pandemic, the odds INPC use was 64% higher at 0 to 2 months post-implementation and 31% higher at >2 months post-implementation compared to pre-implementation (all p-values <0.001). On the other hand, during the pandemic, there was a clear trend of increasing Health Dart effect over time, with the odds of INPC use 21% higher at 0 to 2 months post-implementation (p=0.003) and 46% higher at >2 months post-implementation (p<0.001).
DISCUSSION
This study had two major goals: (1) develop Health Dart, a FHIR app that integrates information from the INPC directly with Cerner in the form of a chief complaint-oriented dashboard; (2) determine how the implementation of Health Dart affected INPC use among clinicians at 14 IU Health EDs using a stepped-wedge trial design. The development of Health Dart addressed two current, major limitations of HIE implementation and use. First, it integrated relevant information from the HIE directly into the EHR as opposed to forcing clinicians to access a separate HIE system. Second, it presented this information in the form of a dashboard focused on a chief complaint, obviating the need for the clinician to manually collate this information.
As a result of the implementation of Health Dart, INPC use increased by 131% (from 3.6% to 8.3%; p<0.001) overall and by 144% (from 3.6% to 8.8%; p<0.001) more than 2 months after implementation. This increase in INPC use in one health system was remarkable considering the 9% increase for EDs across multiple Indiana health systems reported during the period between 2011 and 2017 subsequent to the passage of the 2009 HITECH Act and UI enhancements.44 Our results support the benefits of direct integration of HIE information into an EHR for HIE use. This is consistent with previous findings that aggregation of data from disparate sources into a single view increased the frequency with which ED clinicians accessed HIE by 91.7%.26 In addition, the display of data in a structured, problem-oriented format, such as Health Dart’s chief complaint view, improves efficiency of data retrieval and user satisfaction.28,45,46
Contrary to our prediction, the effect of the app’s implementation on INPC use was not as strong in encounters that occurred after UI enhancements. The UI enhancements appeared to have had a temporal effect on INPC use, with an initial increase in INPC use in the short term (<2 months) followed by a return to pre-UI enhancement INPC use rates in the medium term (>2 months). The gradual effect of Health Dart implementation and UI enhancements on INPC use is supported by the Innovation Diffusion Theory, in which the use of technology increases slowly at first and then rapidly before leveling off.47 The individual, social, and organizational factors outlined in the UTAUT model also can potentially explain why INPC use rates and methods of use differed by ED and by implementation wave. That is, clinicians in each of the EDs have different expectations (e.g. leaders, peers, and organizational and technical infrastructures) that influence their intention and behavior to use Health Dart.40 Variations in rates of INPC use across sites and implementation waves is also consistent with EHR implementation evaluations which note how variability in implementation processes, such as physician training and timing of software updates, influence clinicians’ perceptions about the usability of a newly implemented technology.48 In our study, which focused on implementation context guided by the CFIR model, the implementation factors (i.e., UI enhancements and COVID-19 pandemic) weakened the effect of Health Dart in increasing INPC use.
Moving the Health Dart app toward the top of the Cerner PowerChart toolbar was associated with an increased likelihood of INPC use among ED clinicians. Relocation of the app to a more prominent location presumably increased its usability by decreasing the time required for users to find and use it. This finding is consistent with Fitt’s law and Jakob Nielsen’s usability heuristics for user interface design. Fitt’s law describes the amount of time it takes a user to complete an action as a function of the distance and accuracy of the movement, while Nielsen’s usability heuristics suggest that visibility and prioritization of relevant content increase usability.49,50 Because the toolbar relocation occurred toward the end of the study, we were not able to separate the effects of the Health Dart implementation on INPC use from the effects of the toolbar changes. To reduce confounding of our results, we excluded encounters that occurred prior to Health Dart implementation when evaluating the effect of toolbar relocation on INPC use.
Congruent with our prediction, the effect of the implementation on INPC use was not as strong in encounters that occurred during as opposed to prior to the COVID-19 pandemic. The pandemic had a temporal effect on INPC use, with a decline in INPC use in the short term (<2 months) followed by a return to pre-COVID-19 INPC use rates in the medium term (>2 months). The results of the COVID-19 modeling should be interpreted with caution as the lower volume of patients and higher rate of infectious diseases and respiratory conditions in encounters during the pandemic increase the potential for confounding.
This study extends our prior work through a larger sample size, increased statistical power, and the ability to limit the effect of confounding using the stepped-wedge trial design.25,44 Although the data were derived from only one statewide hospital system in the Midwest, this novel method can potentially be scaled to additional EDs and HIEs to improve generalizability of results for different populations, hospital systems, and EHR vendor platforms. The comprehensive and diverse data sources of the INPC also serve to improve the generalizability of our results. The INPC connects 123 hospitals from 38 health systems, 19,095 clinical practices and 54,107 providers, and represents more than 19 million patients and 16 billion clinical data elements.51 In current research, we plan to analyze the probability of INPC use given clinician, patient, and encounter characteristics; the utility of INPC access given various encounter, patient and clinician characteristics; and how clinicians decide to access Health Dart, CareWeb or both. A more granular level of analysis may elucidate anomalies, such as IU Health Morgan’s higher rates of INPC use compared to other Wave 4 sites. Additionally, because Health Dart was designed for seven chief complaints, we will examine whether there is a relationship between chief complaint and use of INPC. The user log data indicate if a user accessed INPC data, but to date we are unable to account for what information the user accessed and whether it was relevant to clinical decision-making, care provided or clinical outcomes. Knowing what the user accessed could inform whether the actual rates of INPC use reflect the opportunities for appropriate use. Currently, how to determine optimal levels of HIE use in the ED remains unknown.
Limitations
Similar to other evaluations of healthcare technology adoption and use, this study did not measure the variability among individual clinicians when assessing the causal effect of the Health Dart implementation on INPC use.18-20 Models controlled for ED sites to account for differences between individual clinicians, and we assumed that clinicians working at the same ED had similar organizational and social influences.
CONCLUSION
Our results provide support for the benefits of directly integrating HIE information into the EHR in a problem-oriented format for promoting HIE use in the ED. In addition, they provide evidence of the influence of implementation context in the adoption and use of HIE. The results underscore the importance of considering contextual influences such as culture, policy, and setting when evaluating the implementation of a novel technology in the ED.
Data Availability
All data produced in the present study are available upon reasonable request to the authors
DECLARATION OF COMPETING INTEREST
Authors report no conflicts of interest.
FUNDING AND SUPPORT
This research was made possible by the Lilly Endowment Inc. Physician Scientist Initiative; Indiana University Health and the Indiana Clinical and Translational Sciences Institute, funded in part by grant ULI TR002529 from the National Institutes of Health, National Center for Advancing Translational Sciences, Clinical and Translational Science Award; and the Advances in Medicine (AIM) grant from Cook Medical. Drs. Rivera, Schleyer, Schaffer, and Jang received funding from the Agency for Healthcare Research and Quality under grant R01HS027185. Ms. Hosler is and Dr. Rivera was part of the Public & Population Health Informatics training program at Fairbanks School of Public Health and Regenstrief Institute, supported by the National Library of Medicine of the National Institutes of Health under award T15LM012502. The content of this manuscript is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health, Cook Medical, the Agency for Healthcare Research and Quality, Indiana University, or Regenstrief Institute.
ACKNOWLEDGEMENTS
The authors acknowledge Drs. Richard Holden, Joshua Vest, Julia Adler-Milstein, and Saurabh Rahurkar for their contributions to this research. Thank you to Sarah Zappone, Laura Ruppert, Abena Gyasiwa, and Emily Fortier for their coordination of this work. The authors also thank IU Health, the IU Health ED clinicians, and Indiana Health Information Exchange for their participation in the study.
Footnotes
(hkarns{at}iu.edu)
(jeojang{at}iu.edu)
(jschaffer{at}iuhealth.org)
(pricejoh{at}regenstrief.org)
(schleyer{at}regenstrief.org)
(rerivera{at}iu.edu)
↵1 *CareWeb uses single sign-on through Cerner, typically obviating the need for clinicians to log in separately.