Real World Performance of the 21st Century Cures Act Population Level Application Programming Interface

Objective. To evaluate the real-world performance in delivering patient data on populations, of the SMART/HL7 Bulk FHIR Access API, required in Electronic Health Records (EHRs) under the 21st Century Cures Act Rule. Materials and Methods. We used an open-source Bulk FHIR Testing Suite at five healthcare sites from April to September 2023, including four hospitals using EHRs certified for interoperability, and one Health Information Exchange (HIE) using a custom, standards-compliant API build. We measured export speeds, data sizes, and completeness across six types of FHIR resources. Results. Among the certified platforms, Oracle Cerner led in speed, managing 5–16 million resources at over 8,000 resources/min. Three Epic sites exported a FHIR data subset, achieving 1–12 million resources at 1,555–2,500 resources/min. Notably, the HIE’s custom API outperformed, generating over 141 million resources at 12,000 resources/min. Discussion. The HIE’s custom API showcased superior performance, endorsing the effectiveness of SMART/HL7 Bulk FHIR in enabling large-scale data exchange while underlining the need for optimization in existing EHR platforms. Agility and scalability are essential for diverse health, research, and public health use cases. Conclusion. To fully realize the interoperability goals of the 21st Century Cures Act, addressing the performance limitations of Bulk FHIR API is critical. It would be beneficial to include performance metrics in both certification and reporting processes.


INTRODUCTION
The SMART/HL7 FHIR Bulk Data Access API [1] is designed to enable standardized access to electronic health records (EHRs) on large populations of patients.The API facilitates the "push button population health" thus fostering data-driven innovation on local, regional, and national scales.As of December 31, 2022, support for it is required universally.The 21st Century Cures Act [2] stipulates that certified health information technology must incorporate an application programming interface (API) that provides access to all data elements of a patient's electronic health record, in a manner requiring no "special effort."In the spring of 2020, a regulation-the 21st Century Cures Act Interoperability, Information Blocking, and the ONC Health IT Certification Program [3], were introduced by the Office of the National Coordinator of Health Information Technology (ONC) to govern the API prerequisite, while also ensuring safeguards against information blocking.This rule requires support for the SMART/HL7 FHIR Bulk Data Access API to allow access to patient-level data across a population, bolstering an array of applications across healthcare, research, and public health.Data exchanged are over 100 elements defined by the US Core Data for Interoperability (USCDI) [4] in Fast Health Interoperability Resources (FHIR) format.
To comply with the Centers for Medicare and Medicaid Services (CMS) Promoting Interoperability Programs [5] and avoid a negative payment adjustment, use certified health information technology (IT).Developers of IT get certified by an authorized certification body that tests for technical compliance but does not currently measure API performance.Specifically, the 170.315(g) (10) certification criterion is a federal mandate stipulating that health IT developers must offer standardized APIs for individual services [6] and for population-based .services [7].This criterion is included in the 2015 Edition Base EHR definition, as part of the Cures Act Final Rule [4] .
The Bulk FHIR Access API is new in 2023 and this exploratory work sought to test these early implementations to understand current state and then possibilities as implementations are iterated on over time.Any newly deployed software requires real world testing to benchmark performance.We measured Bulk FHIR Access API performance across a collaborative network of five provider sites using a range of technologies, spanning two EHR vendor products and a health information exchange (HIE) data repository.

METHODS
Participants.We selected five healthcare sites with high performing information technology teams from across the US.All were university-affiliated academic medical centers.The FHIR APIs at three sites were provided by Epic (one was remote hosted) and one by Cerner (also remote hosted) as part of their g (10) certified products.The HIE fashioned its own Bulk FHIR Access API as a facade on top of a local relational database populated by HL7 V2 messages.
Following the Bulk FHIR Access Implementation Guide, the HIE service reads rows from its relational database in bulk, converts them into FHIR resources, writes them into NDJSON files, and returns URLs for downloading them to a final Bulk FHIR polling request upon completion.
Bulk FHIR Testing Suite.We relied on the open source SMART Bulk FHIR Client [1,8], which each performance site downloaded and set up behind their institutional firewall, creating a local implementation.As part of this effort, the Bulk Data Client was instrumented to log the elapsed time for each component of the data retrieval and download workflow.FHIR resource types in scope.The FHIR data format employs a modular approach, breaking down healthcare information into an array of data models called "Resources" that range from medical conditions to medication requests, each with its distinct searchable parameters and governance rules.As a result, every patient's healthcare journey is represented by a unique combination of these Resource types, which can be selectively accessed as needed.We focus on six specific Resource models-Patient, Encounter, Condition, DocumentReference, Observation, and MedicationRequest-that align with the criteria set forth in the USCDI v1 standard and are pertinent to population health scenarios.
Cohort creation.The Cures Act Final Rule [3] requires certified EHRs to support exporting predefined groups of patients through the API described in the Bulk FHIR specification.Notably, the process with which healthcare sites define these groupings of patients within the EHR and the supported group size are not standardized across systems.Cerner currently places a limitation of 20,000 patients per list and suggests groups are kept at or below 10,000 patients for performance reasons.For group attribution in Cerner, the process begins with a query using Cerner's Discern Visual Developer tool.Next, the resulting Cerner PERSON_IDs are transcribed to a spreadsheet, which is uploaded into the Ignite Management Tool.For the duration of this project, the Ignite Management Tool was unavailable; however, the site was able to work with Cerner for assistance.
Epic uses pre-existing registry functionality that does not place a limitation on the number of patient records that can belong to a group but encourages clients to limit requests to groups at or below 1,000 patients.Both Epic and Cerner provide the ability to define patient groups by inclusion criteria such as presence of specific coded diagnoses or having a visit of a specific type within a time window, though the interfaces differ.To compare performance across varied EHR .vendor implementations and varied hospital volumes, a set of adaptable inclusion criteria was selected to define patient groups of varying sizes between 1,000 and 20,000.The HIE has a dedicated Data Core team that provisioned their testing team with a single large list of patients based on a current public health project leveraging Bulk FHIR.A breakdown of the sites and patient groups involved in each test is provided in Table 1. Patients with a positive COVID diagnosis in the county.Exports were done 1 month at a time using "_since" and an experimental "_until" parameter.

Table 1. Description of sites contributing performance data to the study
FHIR API Queries.Each site sought to extract at least one year of data on the populations slated for testing, across six FHIR resource types.Sites were encouraged to limit the observation resource categories to only those for laboratory results, vital signs measurements, and smoking status records.Multiple API parameters were defined in the bulk data access implementation guide, and sites were encouraged to use the "_type" parameter to limit resource type requests.
. Date filtering could be accomplished either by the "_since" bulk API parameter for timestamps or the more flexible "_typeFilter" parameter that enables clients to provide FHIR REST search queries.Uniform date filtering was not required to accommodate variations in Bulk FHIR implementations.The testing plan was circulated in January 2023 to the five participating sites and teams managing access to their Bulk FHIR APIs, with a goal to review results in March.
SMART on FHIR Testing Suite.As an alternative approach to extracting a cohort from the EHR FHIR Bulk Data API, we tested the submission of many repeated requests using the SMART on FHIR API with a "Crawler" client developed for the task that iterated over the given population one patient at a time.Of note, the single patient FHIR API has a different set of configurable search parameters, allowing sites that were not able to use the "_since" or "_typeFilter" parameters to request a more targeted population for comparison.

RESULTS
All sites were able to install local copies of the Bulk FHIR Testing Suite and four of the five sites were able to perform a series of tests between April and August of 2023.The four sites using g (10) certified EHR products each used the vendor provided tools to request groups be provisioned for the purpose of performance testing.Sites encountered some issues and at times substantial delays as they learned to navigate these tools.Time taken to export records by server and method.Logs created by the Bulk FHIR Testing Suite and the SMART on FHIR Testing Suite were analyzed to generate statistics comparing the performance of the local implementations.Statistics include total counts of each resource type able to be downloaded for the cohort, total time taken, and overall number of resources able to be exported and downloaded per minute.
Performance scaling differences between vendors.In this initial series of tests on some of the earliest Bulk FHIR services in production, the Oracle Cerner implementation demonstrated higher overall Bulk FHIR export speeds than the three Epic implementations, and both were slower than the HIE bulk service.Of course, these test sites represent only a small sample of certified implementations from both vendors, and do not permit control for differences among EHR implementations.Tests were run on populations larger than Epic's recommended threshold of 1,000 patients per request.As seen in Figure 1, the custom-built HIE FHIR Service performed very well, averaging over 11,000 resources per minute over its export, and the Oracle Cerner site averaged over 8,000.On these three early Epic Bulk FHIR implementations, export speeds were highly variable on the smaller tests and were observed to approach 2,500 resources per minute on groups larger than 1,000 patients.Discussion with Epic engineers on the performance for larger groups inspired our development and use of the SMART on FHIR test suite that allowed sites to make serial requests "one patient at a time".This method was intended to showcase a baseline of performance and to allow additional filtering options initially unsupported by Epic's Bulk APIs.
Interestingly, early tests showed instances of this approach providing Epic sites faster and more complete access to bulk exports of interest than they could obtain through their Bulk FHIR APIs.
In contrast, the Cerner implementation slowed down when resources were requested one patient at a time with this method.present only in the resource through a reference to another "Binary" Resource.The Binary endpoint containing the clinical note text was not directly tested.
Other filters were employed where supported to narrow the request to only resources of interest.
Using "_since", "_typeFilter", or corresponding date filters from the SMART on FHIR test suite enabled requesting only 1 year of data, which significantly reduced the number of resources in the exports as shown in Figure 3.c.Challenges.The endpoints at Sites 1, 2, and 5 were able to complete exports that included the six resource types using the Bulk FHIR test suite.Sites 3 and 4 (both Epic) encountered errors that required fallback to the SMART on FHIR REST API to export some of the resource types successfully: DocumentReferences and Encounters at Site 3 and DocumentReferences at site 4.
Some errors were encountered that were determined to be caused by data incompleteness or improper template configuration.For example, at Site 1 (Cerner), data elements required by the FHIR API were missing or not represented to the vendor as expected.This prevented the site from reliably exporting all Observation and DocumentReference instances for the cohorts over the study period without falling back to the patient by patient SMART on FHIR REST API to fill in gaps.Over two months, Cerner worked with the site to troubleshoot the issue and provide better error handling, which enabled exports to finish that contained errors, rather than having them fail prematurely and return none of the previously generated data.The limitation that patient groups are capped at 20,000 introduces additional workflow complexity to manage exports over a larger population.Lastly, while Cerner does support date filtering with the "_since" bulk API parameter, early tests showed no decrease in the amount of time the server took to prepare and complete filtered vs unfiltered requests.
Among the initial Epic implementations tested, successfully exporting Bulk FHIR for groups of 1,000 patients or more was difficult, with additional issues being identified as requests scaled.
Several exports ran into errors and had to be aborted after multiple days of processing.After careful configuring, sites 3 and 4 were able to export a subset of resource types for groups of around 5,000 patients.Site 4 worked closely with Epic to investigate issues and discuss the use cases being tested and possible enhancements.They were then able to complete an export request for over 10,000 patients, though the export took nearly two weeks to complete and an expired token prevented downloading the data in the end.

DISCUSSION
In the realm of 21st Century interoperability regulatory science, these measurements represent the first performance evaluations, to our knowledge, of early SMART/HL7 Bulk FHIR API implementations.This study served a dual purpose: it not only provided critical performance metrics but also catalyzed robust discussions and led to tangible improvements in software and workflows among healthcare systems, EHR vendors, and API end users.
Because the custom API implementation at the HIE, which took only a few weeks to design, implement and use, outperformed the g( 10) certified implementations, substantial potential for improvement by EHR vendors is evident.Our experience highlights that for this technology to reach its full potential, and indeed meaningful compliance with the law and regulation, more work is needed on the EHR vendors' Bulk FHIR implementations.ONC sponsored two meetings, one before the Rule's publication and one after [9,10], to identify high value use cases for Bulk FHIR.While the vendors have now stood up and supported initial implementations, real world performance of the APIs is not quite there yet to support many of these use cases.The barriers persist even when EHR vendor support is available to high-performing IT teams at leading care delivery sites.
To address myriad identified use cases in care delivery, value, research and population health, meaningful compliance with the Cures Act will require EHR developers to sustain focus on improving current implementations.In some cases, we found that making thousands of .
individual FHIR requests using the SMART crawler was faster than making a single request for the same data in bulk, which indicates there may be sequential processes in the bulk API response that could be parallelized.Cerner supports the inclusion of parameters in bulk requests to filter and limit the FHIR data being returned.Epic does not yet support these filters, which could improve performance by bypassing the processing of unnecessary results.The highly performant bulk implementation at the HIE, built on their data warehouse, suggests that all EHR vendors could potentially achieve substantial improvements, as Cerner does now, by investing in bulk export capabilities that run on their data warehouses rather than their transactional databases.
With the current performance limitations, large enough datasets for some use cases could take not hours but years to be generated.Although Epic has suggested limiting the use of the Bulk FHIR Access API to 1,000 patients or fewer, we believe that such restrictions are inconsistent with the intent of the API provisions of the Cures Act Rule.Incorporating performance measurements alongside technical compliance to incentivize and track real world usability of the APIs should be considered both in the ONC Health IT Certification Program [11] and the EHR reporting program [12].Yet, interesting challenges lie ahead in designing a performance-based criterion.For instance, being able to export a flat number of records per day won't scale for larger health systems that may have much more patient volume.A specific metric may be needed, for example the ability for health systems to export US Core data for their entire active patient population within a certain time frame, say 5 days.All interested parties-the EHR vendors, healthcare organizations, policymakers-can work together to establish feasible metrics.Partnering in this discussion will allow everyone to collaborate on this imperative technology.
. This study is intended as a contribution to the regulatory science around health IT, informing the development and refinement of interoperability law and regulation.Future investigations could broaden the scope by including additional FHIR resource types, engaging more sites, and incorporating a wider array of vendors.To sharpen the performance assessment, subsequent tests might examine server response times for various request stages, and compare peak and off-peak speed variations.The standout performance at the HIE site underscores a compelling avenue for innovation: the development of population-level Bulk FHIR interfaces.These could synergize with patient data soon to be accessible using the Trusted Exchange Framework and Common Agreement (TEFCA), thereby unlocking new dimensions of data usability and access [13].
To achieve the seamless healthcare data exchange envisioned by the 21st Century Cures Act, where all elements of a patient's record can be accessed across an API with 'no special effort,' a unified and urgent effort from healthcare providers, EHR vendors, and policymakers is crucial.
While testing came with its challenges, the learnings from this effort will allow continued, strong partnerships with certified health information technology vendors as key interested parties work to transform clinical and public health informatics.The experience of our collaborative network reinforces the value the Bulk FHIR API has as a transformative tool. .


Patients with an emergency department visit during the month of January 2023  Patients with an emergency department visit during Q3 2021.Only pediatric population in the test. Patients with an emergency department visit during Q4 2022.Only pediatric population in the test.Site 2 Patients with an emergency department visit during the most recent week  Patients with an emergency department visit during the most recent month Site 3 Patients with an emergency department visit during the most recent week  Patients with an emergency department visit during the most recent month Site 4 Hypertension management Tier 5 cohort  Asthma Utilization Tier 4 cohort  Congestive Heart Failure cohort Site 5 , and seconds needed to export one patient's complete record on average.P = Patients, E = Encounters, O = Observations, C = Conditions, D = DocumentReferences, M = MedicationRequests, R = total Resources.SoF = SMART on FHIR testing suite.

Figure 1 .
Figure 1.Performance scaling differences between vendors.(a) Measured export speed plotted against total number of resources R in each benchmark.(b) Total count of resources exported at each site divided by total export time.Performance is grouped by API service provider.

Figure 2 .
Figure 2. Performance scaling differences between implementations from a single vendor.

Figure 3 .
Figure 3. Effects of bulk API parameters.(a) average export speeds by resource type, seen by altering the "_type" parameter.(b) breakdown of average resource size for each type in the study, in kilobytes.(c) average number of resources of each type per patient in the query.Counts on the left are without use of the "_since" parameter, and counts on the right are from queries using the "_since" parameter to only export data from the last year or as simulated by date filtering in the SMART on FHIR testing suite.

Table 2 .
the certified technology, which took time to report to the EHR developers and for them to identify and implement solutions.A breakdown of all the test exports is in Table2.Benchmark results.Number of resources exported in each test, with measurements of total time, resources per minute It took between two and 119 days (on average, 65 days) for sites to successfully make any Bulk FHIR API requests after providing selection criteria.The sites required coordination with one or more teams outside of their departments or organizations to gain sufficient access to run the test suites.Notably, this was, for most sites, the first project provisioning Bulk FHIR APIs, and new configurations and workflows involving multiple parties needed to be established.Errors were encountered in the tooling .provided by