Abstract
Objective This study compared primary care physicians’ self-reported experiences with Electronic Health Records’ (EHR) interoperability, as reported across three surveys: the 2022 Continuous Certification Questionnaire (CCQ) from the American Board of Family Medicine, the 2022 University of California San Francisco’s (UCSF) Physician Health IT Survey, and the 2021 National Electronic Health Records Survey (NEHRS).
Materials and Methods We used descriptive analyses to identify differences between survey pairs. To account for weighting in NEHRS and UCSF, we assessed the significance of differences using the Rao-Scott corrected chi-square test.
Results CCQ received 3,991 responses, UCSF received 1,375 from primary care physicians, and NEHRS received 858 responses from primary care physicians. Response rates were 100%, 3.6%, and 18.2%, respectively. Substantial and largely statistically significant differences in response were detected across the three surveys. For instance, 22.2% of CCQ respondents said it was very easy to document care in their EHR, compared to 15.2% in NEHRS, and 14.8% in the UCSF survey. Approximately one-third of respondents across surveys said documenting care in their EHR was somewhat or very difficult. The surveys captured different respondent types with CCQ respondents trending younger, and NEHRS respondents more likely to be in private practice.
Discussion All surveys pointed to room for improvement in EHR usability and interoperability. The differences observed, likely driven by differences in survey methodology and response bias, were likely substantial enough to impact policy decisions.
Conclusion Diversified data sources, such as those from specialty boards, may aid in capturing physicians’ experiences with EHRs and interoperability.
Introduction
The United States Centers for Disease Control and Prevention began conducting the National Electronic Health Records Survey (NEHRS) with funding from the Office of the National Coordinator for Health IT in 2012 with the goal of reporting on national rates of electronic health record (EHR) adoption and patterns of EHR use across office-based physicians in the United States.(1) It has been conducted almost annually ever since. Researchers and policymakers have used NEHRS to evaluate and understand how changes in EHR adoption and other healthcare information technologies (HIT) impact clinical practice. More recently, policymakers have used NEHRS to examine office-based physicians’ engagement in electronic exchange of health information and interoperability of EHR systems.(2) Assessing these HIT functions are important to the Medicare Promoting Interoperability Program and the strategic objectives of the 21st Century Cures Act and actively used by policymakers.(3)
Despite the important information it has provided about clinicians’ transition from paper to electronic records, to date, NEHRS has not been statistically compared to other sources of information on physician EHR use. Surveys of physicians and other professions are challenging and often feature low response rates, raising the possibility of considerable response bias. Response rates in recent years of NEHRS have been as low as 25%, perhaps reflecting well-known challenges in recruiting healthcare professionals to complete surveys.(4,5) Researchers analyzing NEHRS do not have the information required to understand the importance of missing information due to a low response rate and potentially non-representative response cohort. It is unclear whether these are actual or merely theoretical limitations of NEHRS.
In this study, we compared responses to three surveys, each intended to gather information on physicians’ use of EHRs but fielded with substantially different strategies: 1) the 2021 NEHRS; 2) the 2022 Continuous Certification Questionnaire (CCQ) from the American Board of Family Medicine (ABFM); and 3) the inaugural version of University of California, San Francisco’s (UCSF) Physician Health IT Survey, which was also fielded in 2022. The NEHRS was a voluntary survey fielded among a relatively small sample with repeated mailing and active searches to identify the address and contact information for targeted respondents. The CCQ was a required component of Family Medicine physicians’ recertification process through ABFM. The UCSF Physician Health IT Survey was a voluntary survey designed to maximize number of responses but not response rates.
In comparing responses, we focused on the characteristics of responding physicians and the domain of interoperability among physicians providing primary care, as this topic and these respondents were common across all three surveys and assessed using comparable questions. We sought to identify trends common to all three surveys, but as importantly, discrepancies that may highlight potential paths for improved data collection. Policies affecting EHR use and implementation aim to improve interoperability which is vital to patient care. Understanding the strengths and limitations of different methods for ascertaining these policies’ impacts on physicians is critical to inform policymakers, regulators, EHR vendors, and health system leaders working to improve various dimensions of EHR functionality.
Methods
Population and Instruments
We compared three surveys covering physicians’ experiences with interoperability in EHRs: the 2021 NEHRS, the 2022 CCQ, and the 2022 UCSF Physician IT Survey (Table 1).
Physician informants asked to complete the NEHRS were selected through a random recruitment process from the American Medical Association and American Osteopathy Association Master Files, stratified by specialty and region.(4) They were initially sent both mail and electronic recruitment letters and, subsequently, mail and electronic versions of the survey. The survey included questions intended to assess eligibility for participation: specifically, participants were required to spend most of their working time providing patient care, not be federally employed, and be less than 85 years of age at the time of the survey. Although NEHRS samples physicians in primary care, specialty care, and surgery, we included only primary care physicians in this analysis and excluded any respondent who said that they do not use an EHR. In the NEHRS public use file, each individual respondent was assigned a sample weight based on region and specialty. These weights were essential to accurately interpreting NEHRS data and were accordingly integrated into the analysis.
Completion of the CCQ has been required of the more than 102,000 family physicians’ participating in continuous certification processes since 2017.(6) As such, it has a 100% response rate. For nearly a decade the ABFM has included questions about EHR Meaningful Use policies and, for the 2022 CCQ, ABFM collaborated with the United States Office of the National Coordinator for Health Information Technology to incorporate questions about EHR use that closely parallel those in NEHRS. The goal of this collaboration was to enable comparison and to evaluate the potential utility of supplementing NEHRS with outside data. All respondents to the CCQ first answered a set of personal, practice, and demographic questions before being randomized to one of two modules on EHR usage, including one on interoperability. Finally, they were randomized to one of five modules that covered topics such as burnout and meaningful use of EHRs. Thus, EHR module cohorts were 50% of the overall number of respondents, and cohorts for other modules were 20%. Only respondents who indicated that they use an EHR system and that they provided direct patient care were included in the analysis. To ensure comparability with NEHRS, we also excluded federally employed physicians.
The UCSF Physician Health IT Survey was initiated in 2022 to collect in-depth information on how information technology is integrated into clinical settings. Researchers used simple random sampling to select 90,000 potential participants with listed email addresses from Definitive Healthcare, a proprietary dataset designed for healthcare analytics. All sampled physicians received 8 emails. 60,000 sampled physicians also received a postcard reminder to complete the survey. 30,000 sampled physicians also received 2 recruitment letters by mail. The UCSF survey recruited physicians of all specialties, but only physicians who used EHRs and who worked in primary care for non-federal employers were included in this analysis.
Statistical analysis
Both NEHRS and the UCSF survey used survey weights with the goal of improving the generalizability of their findings. We therefore used a Rao-Scott corrected chi-square test, which adjusts the chi-square estimate for weights to account for changes in the composition of the population.(7) Since our interests were in identifying specific differences in the surveys, we conducted significance tests in a pairwise fashion across the three surveys.
While the ABFM CCQ and the UCSF survey both based questions on NEHRS, some questions were phrased differently or included different answer options. We did not statistically analyze differences in the responses to questions that we found to be incomparable across surveys, but we did retain them in the tables for reference.
As a supplemental analysis, we also conducted stratified comparisons of especially important indicators of interoperability experience across age groups (less than 50 or 50-plus years), EHR platform (Epic or other), and practice type (private practice or other).
All analyses were conducted in R version 4.1.2, with the Rao-Scott corrected chi-square test implemented in the “survey” package version 4.1.1.(8,9)
Data availability
The NEHRS data used in this research are publicly available through the website of the United States Centers for Disease Control and Prevention.(10) ABFM CCQ data may be accessed for IRB-approved projects subject to the approval of the ABFM Research Governance Board; please contact the corresponding author for details. The UCSF survey data underlying this article cannot be shared due to privacy restrictions.
Results
Response Rate Comparison
The CCQ had the highest number of respondents and response rate across the three surveys. A total of 3,991 respondents to the 2022 CCQ provided direct patient care and were otherwise eligible for inclusion in the analysis (100% response rate). Of the 10,302 physicians sampled for NEHRS in 2021, 1,875 were eligible and completed the survey (18.2% response rate); although we could not determine how many family physicians were in the NEHRS sample, 858 (48.7% of respondents) listed primary care as their specialty.(4) Among the 90,000 physicians sampled for the UCSF Physician Health IT Survey, 3,209 responded (3.6% response rate), among whom 1,375 (42.8% of respondents) practiced in primary care.
Respondent Demographics
Respondents to the three surveys represented different physician demographics (Table 2). Respondents to the CCQ were the youngest: the share of respondents aged 35 to 44 (42.1%) was over twice that seen in the NEHRS (17.0%) and the UCSF survey (15.2%). Male and female genders were approximately equally represented in all surveys, although women outnumbered men (51.7% to 48.3%) in the UCSF survey.
The share of responses by setting varied across the three surveys. A disproportionate share of NEHRS respondents worked in private practice (73.1%) compared to CCQ (39.8%) or UCSF (29.3%). While a plurality of CCQ respondents were from health systems (46.2%), this setting was not well represented in either the USCF (7.4%) or NEHRS (11.5%) surveys. Far more respondents to the UCSF survey – 29.2% – practiced in academic health centers or faculty practices.
There was also some variability by practice size. Physicians from practices with more than 10 physicians were overrepresented in the UCSF survey (45.1%) versus NEHRS (28.7%). The CCQ asked about the number of providers working at the respondent’s main practice site instead of physicians at all locations of the practice, which is how NEHRS and UCSF surveys asked the question. Despite this more inclusive clinician language, a larger proportion of CCQ physicians (42.4%) indicated that they work in practices with 1 to 5 providers. Geographic representation was similar in the CCQ and NEHRS, but respondents to the UCSF survey were more likely to practice in central metropolitan locations compared to the other surveys’ respondents.
EHR Vendor
There were significant differences in the EHR platform used by respondents and their satisfaction with these EHRs across the surveys (Table 3). While Epic was the most common EHR platform in all three surveys, Cerner was the second most commonly used platform in the UCSF survey and eClinical Works in the others. The CCQ had the largest share of respondents indicating that they do not know what EHR they use (1.8%). Respondents to NEHRS reported being very satisfied with their EHR at significantly higher rates (29.1%) than in the CCQ (26.7%) or UCSF survey (19.4%).
EHR Use and Interoperability
Similarly, documentation and ease of documentation differed (Figure 1, Supplementary Material I). CCQ respondents reported spending more than 4 hours in after-work documentation at significantly higher rates (17.1%) than respondents in the other surveys (12.2% for UCSF and 8.4% for NEHRS). Interestingly, more CCQ respondents said that their experience of documenting care in the EHR was excellent / very easy (22.2%) versus the others (14.8% for UCSF, 15.2% for NEHRS).
Comparison of documentation and practice patterns across the three surveys. For C, note that the CCQ and UCSF survey ask about telemedicine in the prior 3 months, while NEHRS asks about telemedicine since March 2020.
Several questions related to interoperability were similar across all three surveys; however, response options were identical for only two questions assessing two items on physicians’ experience with interoperability (Table 4). 45.8% of respondents to the CCQ and 45.4% of respondents to NEHRS indicated that their EHRs integrate patient health information from outside organizations into their EHR. 22.8% of respondents in CCQ and 21.9% of respondents to NEHRS indicated that they often had access to clinical information from outside organizations in their EHR. On both items, respondents to the UCSF survey were substantially less positive, with 37.6% of UCSF respondents indicating that their EHR integrated information and 14.6% indicating that they often had access to clinical information.
Response options differed between NEHRs and the CCQ and UCSF survey for two additional interoperability options, making direct comparison across the three more challenging. For one of these items, respondents to the CCQ had a better experience of interoperability than respondents to the UCSF survey: 42.3% of CCQ respondents said that they often receive external information via EHR or web portal, compared to 21.2% of UCSF respondents. 64.6% of NEHRS respondents said that they received information but were not asked how often. For the second item, responses to the CCQ and UCSF were more similar: 40.0% of respondents to the CCQ indicated that they often queried for information from outside organizations for new patients compared to 38.1% of respondents to the UCSF survey. 54.4% of respondents to the NEHRS indicated that they queried for information, but again were not asked how often.
Some additional questions about the ease of using interoperable data were only asked in the CCQ and UCSF survey (Supplemental Material II). As before, CCQ respondents report better experiences with interoperability than the UCSF respondents do. For example, 24.1% of CCQ respondents say it is very easy to include external information in care decisions compared to 18.6% of UCSF respondents.
Stratified Comparisons Across Surveys
We designed our stratified analysis to identify whether differences in survey composition — such as a greater number of CCQ respondents in the youngest age group — explained observed differences in opinions or if these differences persisted within sub-groups (Figure 2, Supplemental Material III). Results indicated that survey differences persisted across sub-groups: for instance, both older and younger respondents to the CCQ were more likely to say that information was integrated into their EHR than were the same age group respondents to the UCSF survey.
Stratified comparisons of key interoperability and EHR satisfaction measures
Differences between sub-groups were directionally consistent across most comparisons in the three surveys. For instance, all three found that Epic users were more likely to be very satisfied with their EHR, that physicians in private practice were less likely to have external patient health information available, and that physicians aged 50 years or more were less likely to integrate external information into care decisions.
Discussion
These findings reflect a rare opportunity to compare primary care physician responses related to the aims of EHR policies across multiple surveys with different sampling strategies. The comparison of these three surveys expands our understanding of their validity, reliability, and generalizability. It also aids in determining whether improvements can be made to the current process of collecting information about interoperability and other dimensions of clinician experience with HIT. All three surveys revealed substantial opportunity for improvement in physician experience with HIT and particularly with interoperability. At the same time, the surveys responses significantly differed in several ways, suggesting that policymakers may benefit from integrating sources of information outside of NEHRS.
The surveys were unanimous in showing that physicians were not highly satisfied with EHRs, and that interoperability remains a significant problem. Approximately one quarter of physicians in all surveys indicated that they were very satisfied with their EHR platform. Similarly, approximately three-quarters of respondents reported documenting care for an average of one or more hours after work per day. Across surveys, only about 20% of physicians reported that information was often available from their patients’ healthcare encounters outside the primary care clinic. The three surveys indicated too that users of Epic were more likely to have this information available and that physicians working in private practice were less likely to.
Despite these common findings, almost all questions had statistically significant differences in the responses to the surveys. This suggests that the results of regression-based analyses would differ based on which of these data sources the researchers used, which may have substantial policy impacts. The observed differences likely arose in part from differences in respondents. The ABFM CCQ is a cross-sectional census with 100% response rate but from a single, large specialty. The plurality of CCQ respondents were aged 35 to 44, younger than in the other surveys. They were also over four times as likely to work at a health system compared to UCSF and NEHRS respondents. NEHRS may also have sampled from or had a response bias favoring solo practices and small clinics at a higher rate than the other surveys. However, substantial differences persisted in cross-survey comparisons among physicians who seemed similar.
Response bias likely played an important role in these observed differences, as the UCSF survey and NEHRS had much lower response rates than the CCQ with its 100% response rate. Some of the differences point towards the CCQ’s unique capture of data from respondents whose experiences are especially important to decisions in HIT policy. The response rate comparisons show that physicians are less likely to respond to discretionary, voluntary surveys such as NEHRS and UCSF, potentially making them less representative of the physician population at large. Our findings show that differing survey composition has downstream effects on the results and also indicate that there are likely differences across surveys not accounted for by observable differences in the composition of respondents. At the same time, the CCQ’s response rate, achieved by tying survey completion to the recertification process, may have a downside inasmuch as respondents may be more motivated to provide quick rather than thoroughly considered answers to survey questions.(11)
Physicians in primary care spend more time documenting care than other physicians, so it is vital to have high quality data sources about how they use EHRs.(12) In particular, it is important to find policies that maximize the benefits of EHRs while minimizing their potential to add to physicians’ burdens.(13,14) Our findings suggest that data collected by certifying boards, such as ABFM, have several appealing characteristics that could make them useful as a supplement to NEHRS. First, the data are likely to have a higher assurance of representativeness and reliability, especially if collected as a mandatory portion of recertification activities. Next, these certifying boards could collect data that are more focused on the EHR experience of physicians within particular medical specialties. Certifying boards could also leverage their relationships with physicians to collect ongoing qualitative information about emerging issues with EHRs and other medical technologies that can inform future surveys. However, individual certifying boards will by definition only be able to represent the views of an individual primary specialty and any related subspecialties.
Limitations
There were several limitations to this work. Despite coordination between the three institutions, there were some differences in the questions or their response options. Some questions, such as the question about practice size, had been carried forward from previous versions of the CCQ. This limited our ability to reach more detailed conclusions about some potential comparisons. Perhaps more substantially, the CCQ only included physicians within one subspecialty of primary care, albeit the third largest medical specialty. There are some differences between family physicians and other types of primary care physicians, including that family physicians are more likely to practice in rural and underserved areas.(15) In many other ways, though, family physicians are quite similar to other physicians working in primary care.(16,17) Without information from other primary care specialties, it is impossible to know what impact this has on our conclusions.
Conclusion
Our study compared the NEHRS, a key national survey used to inform policymakers on the state of HIT, with other surveys and found that while similar conclusions could be drawn at a high level related to the state of interoperability and physicians’ experiences documenting in EHRs, there were also significant differences when looking at the findings at a more granular level. There were also important differences in the composition of the respondents which likely impacts the results of these surveys. These differences are meaningful for policymakers whose objectives are to advance the development and use of HIT and improving data sharing in service to patient care. Future work is needed to assess variation in self-reported HIT experiences of physicians in other medical subspecialties. The strategy of augmenting NEHRS with data from the recertification surveys of other certifying boards would be extremely valuable for the high reliability they offer.
Funding
Office of the National Coordinator for Health Information Technology, Department of Health and Human Services, Cooperative Agreement Grant # 90AX0032/01-02
Supplemental material for “Comparative Analysis of Three Surveys on Primary Care Providers’ Experiences with Interoperability in Electronic Health Records”
I. Documentation and visit practices
II. Integration into care of patient information from external organizations
III. Stratified comparisons
Acknowledgments
We gratefully thank the respondents of all surveys for their participation.
