You are here

Article Text

Article menu

Download PDFPDF

Original research
When order sets do not align with clinician workflow: assessing practice patterns in the electronic health record
  1. Ron C Li1,
  2. Jason K Wang1,
  3. Christopher Sharp2,
  4. Jonathan H Chen1
  1. 1 Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California, USA
  2. 2 Stanford Health Care, Stanford, California, USA
  1. Correspondence to Dr Ron C Li; ronl{at}stanford.edu

Abstract

Background Order sets are widely used tools in the electronic health record (EHR) for improving healthcare quality. However, there is limited insight into how well they facilitate clinician workflow. We assessed four indicators based on order set usage patterns in the EHR that reflect potential misalignment between order set design and clinician workflow needs.

Methods We used data from the EHR on all orders of medication, laboratory, imaging and blood product items at an academic hospital and an itemset mining approach to extract orders that frequently co-occurred with order set use. We identified the following four indicators: infrequent ordering of order set items, rapid retraction of medication orders from order sets, additional a la carte ordering of items not included in order sets and a la carte ordering of items despite being listed in the order set.

Results There was significant variability in workflow alignment across the 11 762 order set items used in the 77 421 inpatient encounters from 2014 to 2017. The median ordering rate was 4.1% (IQR 0.6%–18%) and median medication retraction rate was 4% (IQR 2%–10%). 143 (5%) medications were significantly less likely while 68 (3%) were significantly more likely to be retracted than if the same medication was ordered a la carte. 214 (39%) order sets were associated with least one additional item frequently ordered a la carte and 243 (45%) order sets contained at least one item that was instead more often ordered a la carte.

Conclusion Order sets often do not align with what clinicians need at the point of care. Quantitative insights from EHRs may inform how order sets can be optimised to facilitate clinician workflow.

  • decision support, computerized
  • information technology
  • near miss
  • medication safety
  • computerized

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjqs-2018-008968

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    Billions of dollars have been invested into electronic health records (EHRs) with the promise of improving quality by increasing workflow efficiency and reducing errors and unwanted care variability, but the effects have been mixed and not well understood.1 2 Clinical decision support (CDS) refers to tools such as computerised alerts and condition-specific order sets that are intended to intelligently filter information to the clinician to enhance workflow and decision-making. The ‘CDS Five Rights’ is a framework to guide the appropriate design of CDS to communicate the right information to the right person in the right CDS intervention format through the right channel at the right time in workflow.3 Nevertheless, CDS often falls short of these goals in practice, in part due to the difficulty in anticipating how the complex environment of clinical medicine affects clinician workflow. Poorly designed CDS may in fact hinder care delivery by creating new unintended downstream hazards due to factors such as poor user interface design and introduction of new complexities in workflow.4–6 For example, many CDS tools face issues such as alert fatigue with high override rates7 8 or automation bias where clinicians over-accept computer-generated output in EHRs “as a heuristic replacement of vigilant information seeking and processing” even if the recommendations are inappropriate.9 Purposeful and safer CDS design requires that we understand the gaps between the expectations and realities of how CDS impacts clinician workflow in the real-world setting.

    Order sets are collections of clinically related items grouped together for a wide array of specified clinical scenarios and constitute a significant component of EHR order entry.10 11 Order sets intend to help clinicians more effectively access appropriate items compared with individual ‘a la carte’ order entry, thereby reducing undesirable care variability12 and improving adherence to evidence-based practices.13–16 The extent to which order sets actually support clinician workflow is not well understood. The large number of order sets that healthcare institutions have to build and manage make it difficult to systematically evaluate whether the content and design for each order set optimally aligns with clinician needs.17 Poorly built order sets with outdated content can clutter the ordering interface, create distractions or even prompt clinicians to order unnecessary items due to automation bias.18 Although federal requirements exist for regularly scheduled order set reviews,19 institutions typically rely on a ‘top down’ approach of employing small groups of clinicians to curate order sets based on clinical guidelines, institutional policies and expert opinion with limited quantitative insight into how the order sets are actually being used by clinicians.20 Consequences of suboptimally designed order sets may remain unnoticed, perhaps not until an adverse event serious enough to warrant institutional attention.21 Continuous improvement in the quality and safety of health information technology design requires an understanding of how tools like order sets are used in real-world settings.22

    We can gain unique insights into end-user experience through implicitly crowdsourced experience derived from data in the EHR audit trail. EHR systems log the timestamps of all transactions that can reflect in situ clinician behaviour without the biases associated with purposefully collected user feedback.23 This type of data has been used to analyse clinician practice patterns in order to derive clinical pathways,24 create recommender systems that suggest commonly entered orders for certain clinical scenarios25–27 and detect incorrect orders in the EHR.28–30 In our study, we seek to analyse order set usage at an academic medical centre to assess alignment with clinician needs. We focus on examples of poor alignment such as bloated order sets, whose items are rarely used or of limited use, as metrics for order set optimisation.31 We assess potential shortcomings that interfere with workflow, such as poorly accessible order set items that prompt users to instead order them a la carte, or the risk of automation bias that prompts clinicians to order items in error. We specifically look for four indicators in the EHR: (1) infrequent ordering of order set items, (2) rapid retraction of order set items, (3) additional a la carte ordering of items not listed in an order set and (4) a la carte ordering of items despite being listed in an order set.

    Methods

    The following terminology will be used throughout this paper: an ‘item’ is something that can be ordered in the EHR (eg, a medication or laboratory test), an ‘order set item’ refers to an item listed in a specific order set (eg, the same medication in two different order sets would be considered two different order set items), an ‘order’ is a single transaction of ordering an item, an ‘order set order’ is an order originating from an order set, an ‘a la carte order’ is an order that did not come from an order set and an ‘order set use’ refers to an instance of an order set being used to order at least one item. We mined data from all inpatient orders of medication, laboratory, imaging and blood product items along with all order set uses in the EHR from 1 January 2014 to 15 July 2017 at a 613-bed tertiary care teaching hospital. Laboratory tests ordered together in a panel (eg, complete blood count) were counted as one individual order. Medications were normalised by their formulation and route of administration. During this time, order sets were created and maintained by a standing order set review committee that met every 3 months to review one to two order sets at a time using standards derived from the Institute for Safe Medication Practices32 as well as input from clinicians and clinical informaticians at our institution. An order set was typically reviewed once every 3 years. All clinicians practising in the inpatient setting had access to the entire list of order sets through the EHR, although training and awareness of each order set varied by provider specialty. All data were de-identified and extracted via our institution’s clinical data warehouse.33 We then assessed the following indicators of poor alignment:

    Indicator 1

    Infrequently ordered items were identified by ranking the ordering rate of order set items, which equals the number of orders for a given item divided by the respective number of order set uses.

    Indicator 2

    Rapidly retracted orders were defined as orders that were discontinued within 30 min, based on prior studies establishing these as surrogates for erroneous orders.28 30 Only medication orders were included in this analysis due to availability of data at the time of this study. For a given item, the order set retraction rate equals the number of retracted orders from a given order set divided by the total number of orders of the item from the order set, while the a la carte retraction rate equals the number of retracted a la carte orders divided by the total number of a la carte orders of the item. The relative risk of retraction of an order set item was calculated by dividing the order set retraction rate by the a la carte retraction rate.

    Indicator 3

    We employed an itemset mining approach25 26 34 to look for a la carte orders co-occurring within 10 min of a given order set use. These ‘additional a la carte orders’ represent orders of items that were not part of the order set, but were deemed necessary by the clinician at the point of care when using the order set. We chose our window to be 10 min a priori based on our experience using order sets in the clinical setting. For each order set, we identified such items with orders that co-occurred at a rate greater than the median ordering rate of the items in the order set. The co-occurring rate of an item–order set pair was calculated by dividing the number of times a given item co-occurred at least once by the total number of order set uses for that given order set. We also filtered for orders of items that were significantly more likely to co-occur with a given order set than with all order sets. We excluded the item ‘point of care glucose’ from the filtered results because it is a non-specific item commonly ordered a la carte by nurses.

    Indicator 4

    We identified a la carte orders co-occurring within 10 min of a given order set use that were of items included in that order set. These ‘a la carte over order set’ orders were of items ordered a la carte despite being listed in a given order set. We filtered for items that were ordered a la carte more frequently than the ordering rate of that item from the order set and calculated the a la carte to order set ordering rate ratios for each item–order set pair.

    Tests for significance for all analyses were performed with Fisher’s exact tests at Bonferroni-corrected p-value thresholds.35

    Case study

    We conducted focus group interviews with five members of our institution’s inpatient order set review committee that explored how the indicators derived from this study could be incorporated into the order set review process. We then presented an example of our analysis for the inpatient congestive heart failure (CHF) order set to the committee while it was undergoing review as part of an effort to improve adherence to an evidence-based inpatient CHF clinical pathway.36

    Results

    Between 1 January 2014 and 15 July 2017, a total of 17 072 459 medication, laboratory, imaging and blood product orders were generated for 77 421 inpatient encounters. There were 11 108 distinct items, ranging from four blood product items to 8368 medication items (table 1). Moreover, 1785 of these items were available in the 545 order sets used during this period in the form of 11 762 distinct order set items, which produced 2 449 208 medication, laboratory, imaging and blood product orders. Among these order set orders, medications were the most commonly ordered items (59%) while imaging orders were the rarest (2%).

    View this table:
    Table 1

    Breakdown of the number of items and orders by item type

    Indicator 1: infrequently ordered order set items

    We first examined the distribution of ordering rates of all 11 762 order set items. The ordering rates varied greatly, ranging from 0.001% to 100% with a median of 4.1% (IQR 0.6%–18%). Laboratory items tended to have higher ordering rates (median 11.5%, IQR 2.5%–33.3%) than medication (median 2.3%, IQR 0.4%–1.1%), imaging (median 4.7%, IQR 0.8%–14.6%) and blood product items (median 2.6%, IQR 0.3%–24.5%) (p<0.001; online supplementary material 1). The order set items with the highest and lowest ordering rates are shown in table 2.

    Supplemental material

    View this table:
    Table 2

    Order set items with the highest and lowest ordering rates, excluding uncommon order sets used fewer than 100 times

    Indicator 2: rapidly retracted orders from order sets

    We next focused on the retraction of medication items, as defined by medications ordered and discontinued within 30 min. At least one medication order was retracted in 13 769 (18%) patient encounters. Among all 1 441 338 medication orders from order sets, 32 072 (2.2%) were retracted, which was lower than the retraction rate for the 3 273 894 a la carte medication orders (3.9%) (p<0.001). Among the 2606 order set medication items that produced at least one retracted order, however, the order retraction rates varied greatly (range of 0.1% to 100% with a median of 4% (IQR 2%–10%)). Specifically, we evaluated whether certain order set medication items were more likely to be retracted than if they were ordered a la carte. One hundred forty-three (5%) order set medication items were significantly less likely while 68 (3%) were significantly more likely to be retracted than if the same medication was ordered a la carte (at a p<2×10−5 Bonferroni-corrected threshold). The order set medication items with the highest and lowest relative risks of retraction are shown in table 3.

    View this table:
    Table 3

    Order set items with the highest and lowest risks of order retraction relative to when the same items were ordered a la carte, excluding uncommon order sets used fewer than 100 times

    Order set items with the lowest relative risks, such as lidocaine from the Bronchoscopy order set and heparin from the Haemodialysis order set, tended to be medications part of well-defined protocols (eg, lidocaine and heparin are routinely given at the onset of bronchoscopies and haemodialysis sessions for local anaesthesia and prevention of clotting in the dialysis circuit, respectively) in order sets that were specifically built for those protocols. Conversely, items with the highest relative risks, such as intravenous esmolol from the Vascular ICU order set and atorvastatin PO from the Neurology General Ward Admit order set tended to be part of larger order sets with broader clinical use cases. These medications may be relevant for some, but not all clinical situations that their respective order sets are used for. The distribution of relative risks of all order set medication items are shown in online supplementary material 1. There was a negative association between ordering rate and order retraction rate. Order set medication items with ordering rates in the lowest quartile were approximately twice as likely to be retracted than those in the second to fourth quartiles (p<0.001) (figure 1).

    Figure 1
    Figure 1

    Relative risk of order retraction of order set items by ordering rate. Items with higher ordering rates were less likely to be retracted than those in the lowest ordering rate quartile (p<0.001).

    Indicator 3: additional a la carte orders of items not listed in order set

    When using an order set, clinicians may order additional items a la carte if they are also needed at the time but not available in the order set. We looked for such items that were additionally ordered a la carte within 10 min of an order set use more often than the median ordering rate of items in the corresponding order set. Of the total 545 order sets, 214 (39%) had such an item commonly added a la carte, with a median of 4 (IQR 1–12) distinct additional a la carte items per order set. The median ordering rate of these additional items was 2.3% (IQR 1%–6%). The items with the highest additional a la carte ordering rates are listed with their corresponding order sets in table 4a.

    View this table:
    Table 4

    (a) Order set items with the highest additional a la carte ordering rates, excluding uncommon order sets used fewer than 100 times (these are items that clinicians frequently needed at the time of order set use, but were not included in the order set build), and (b) order set items with the highest a la carte to order set ordering ratios, excluding order sets used fewer than 100 times (these are items included in order sets, but were frequently skipped over by clinicians and instead more likely to instead be ordered a la carte right after order set use)

    Indicator 4: a la carte ordering of items despite being listed in order set

    When using order sets, clinicians may instead order a listed order set item a la carte because they do not see the item in the order set or purposely choose a la carte ordering because it is more convenient. We quantified how often this phenomenon occurs by looking for items that were ordered a la carte within 10 min of an order set use more often than directly from the order set. Of the 545 order sets, 243 (45%) contained at least one such ‘a la carte over order set’ item, with a median of 4 (IQR 2–9) distinct items per order set. The median ordering rate of these items was 1% (IQR 0.2%–3%). We further identified a subset of order set items for which the a la carte ordering rate was significantly higher than the order set ordering rate using a Bonferroni-corrected threshold of p value <2.5×10−5. In this subset, there were 73 order sets, a median of two distinct items per order set (IQR 1–6), with a median a la carte ordering rate of 2% among all the items. The median ratio of the a la carte ordering rate over order set ordering rate of these items was 8.9 (IQR 3.9–21.5). The order set items with the highest a la carte to order set ordering rate ratios are listed in table 4b.

    Case study: assessing the inpatient CHF order set

    We present several examples of how data-driven insights derived from the above ordering patterns were used to guide the review of the CHF order set by our institution’s order set committee. This order set was built to support both the floor admission process as well as medical decision-making around CHF management. Sections and subsections of the order set are listed in online supplementary material 1.

    The committee was unanimous in agreeing that all four indicators of workflow alignment provided by our study added value beyond expert opinion for order set review. The following findings were unexpected and of particular interest to the order set committee: (1) the items with the highest ordering rates were general hospital admission laboratories (eg, basic metabolic panel (56%), complete blood count (52%)) while most CHF-specific items were relatively infrequently ordered (furosemide (10%), echocardiogram (10%) and NT-proBNP (9%) were the only CHF-specific items with ordering rates above 5%); (2) standard CHF medications such as carvedilol and lisinopril had very low ordering rates below 1% but relatively high retraction rates above 5% from the order set; (3) oral aspirin, which was not included in the order set, was added as an a la carte item 8% of the time the order set was used; and (4) most of the CHF medications listed in the order set were more likely to be ordered a la carte despite being available as order set items. The order set committee subsequently added oral aspirin to the order set and are now discussing ways to further separate CHF-specific medications from general inpatient admission items in the order set. The full list of indicators for the CHF order set items are in online supplementary material 1.

    Discussion

    We present quantitative evidence from EHR usage data that order sets often do not align with what clinicians need at the point of care. Misalignment was reflected in the infrequent ordering of order set items, retraction of medication orders from order sets and frequent a la carte ordering of additional items regardless of whether they were present in the order set. This gap between the assumptions and realities around order set use reflects an opportunity to use real-world usage patterns for optimisation.

    We found significant variability in alignment across order sets. Certain order sets appeared to meet clinician needs well. These order sets contained items that were frequently ordered with low retraction rates, with some items having even lower retraction rates when ordered from the order set than if they were ordered a la carte, suggesting the order set content is well suited to the corresponding workflow. Yet, half of all order set items were rarely ordered (less than 4% of the time). This suggests that many order sets may be bloated with low yield items. These rarely used items were also more likely to be rapidly retracted, indicating they may predispose users to order items by mistake. We identified multiple medications that were more likely to be retracted from certain order sets than when ordered a la carte, suggesting automation bias when clinicians are exposed to these medication–order set combinations. Moving beyond anecdotes of order sets promoting wrong medication orders,21 our analysis provides more explicit evidence of such phenomenon.

    Many order sets may also be missing items that users need. Of the order sets at our institution, 45% were associated with at least one additional a la carte item that was ordered more frequently than half of the items actually included in the order set. Table 3a illustrates many clinically reasonable additions to their corresponding order sets that were not included in the initial build. This is likely because their relevance was not immediately obvious to the order set authoring committee. For example, ondansetron is an antiemetic medication that does not directly treat pneumonia or chronic obstructive pulmonary disease (COPD) and thus was not included in those order sets. However, we see that clinicians were often adding ondansetron as a la carte orders, most likely done to treat nausea, a common symptom among patients presenting with acute pneumonia or COPD. Similarly, we identified aspirin as a common additional a la carte item for the CHF order set. While aspirin is not specifically a CHF medication, it is commonly appropriate for patients with CHF due to comorbid coronary artery disease. This was compelling enough for our local order set committee to add aspirin as an option to the CHF order set. Other examples from table 3a, such as the basic metabolic panel and complete blood count without differential, were most likely added because they are typically preferred by clinicians over the comprehensive metabolic panel and complete blood count with differential, two items that are in the pneumonia order set, because the latter panels contain additional tests that are usually unnecessary for pneumonia management.

    These indicators need to be considered in the context of the clinical use cases specific to each order set item. For example, while a low ordering rate may indicate certain order set items are no longer relevant and should be removed, this may not be true for other infrequent items that are purposely included for rare but serious clinical situations that necessitate prompt action. Similarly, items added a la carte at a high rate should not necessarily be included in the preceding order set if they are not topically related. These trade-offs should be carefully evaluated with the appropriate stakeholders familiar with each unique clinical workflow.

    Our findings also suggest that order sets may sometimes hinder order entry. We found clinicians skipping over certain order set items, only to order them a la carte immediately after. This phenomenon may reflect shortcomings in the order set build itself rather than the clinical appropriateness of the item. For example, either clinicians did not find the item in the order set or they purposely preferred to order the item a la carte because it was more convenient. The degree of preference for a la carte ordering is striking among the order set items in table 3b. For example, of the 17 250 times the orthopaedics inpatient joint replacement order set was used, intravenous hydromorphone was ordered directly off of the order set only once (0.005%), but ordered a la carte within 10 min of order set use 1043 times (6%). A closer inspection revealed that accessing intravenous hydromorphone from the order set required multiple clicks by first checking the ‘severe pain’ box in the ‘opioid analgesics’ section, which only then displayed the option to order the medication. Our results suggest that while this build feature does deter ordering of intravenous hydromorphone from the order set, clinicians develop a workaround and order it a la carte anyway.

    This study raises the question of whether order sets are the right channel to deliver otherwise clinically appropriate content at the right time in workflow. For example, we found that the medications specific to an intended CHF clinical pathway were rarely ordered from the CHF order set. One explanation is that this order set also includes standard admission orders that are pre-selected when the order set is launched. Grouping these items with CHF medications may not be effective because clinical decision-making around administering disease-specific medications often occurs at a later time in workflow after processing standard admission protocols. Further, our EHR design precludes the user from viewing data such as vitals, laboratory results and home medications simultaneously while selecting items from order sets. This design limitation may explain why certain order set items, such as those featured in table 4b, tend to instead be ordered as a la carte items. Reconciled home medications are ordered as a la carte orders in our EHR, which may also explain why common outpatient CHF medications were rarely ordered from the CHF order set. We observe that the most commonly ordered items from the CHF order set were non–CHF-specific standard admission orders, suggesting that the CHF order set was in fact mostly used to access generic admission orders rather than items specific to a CHF clinical pathway.

    While several recent studies have proposed different methods for data-driven order set construction using EHR usage data,25 27 31 our study applies additional methodologies to reveal historical population-level order set usage trends that inform several design considerations. Frequently ordered, rarely retracted items tend to be part of order sets for single, well-defined clinical processes with protocolised workflows. Order sets constructed for individual clinical protocols may be particularly well suited for clinician workflow and reduce practice variability. Conversely, it is less clear whether order sets that are intended to support multiple clinical processes involving more complex medical decision-making successfully achieve that purpose, especially if these clinical processes are meant to take place over different points in time. Items from disease-specific addendums to admission order sets, such as antibiotics for pneumonia, were rarely ordered and in fact were often quickly retracted if ordered. Our study suggests that while inpatient admission order sets may work well for standard general admission protocols, embedding additional condition-specific items may not be effective in supporting management of those conditions, especially if multiple conditions associated with different types of patients and workflows are represented in the order set. Alternate tools that enable more dynamic support of clinical workflow and decision-making needs that evolve over time may be required.

    There are several limitations of this study. Our analysis was confined to inpatient orders from a single academic medical centre, so the order set usage patterns we observed may not generalise to other practice settings, although the four indicators we propose could be reproduced to evaluate any local order set usage. Changes in order set design and policies may confound metrics such as ordering rates of items, which were calculated with the assumption that all items were available in the order set through the entire study period. Given the order set review schedule at our institution, however, we found minimal changes made to most order sets during our study period. We were not able to distinguish between different ordering providers in our analysis, although it is common practice at our institution for only a single primary inpatient provider to enter orders for a given patient. Our relative risk analysis of order retraction between order set and non-order set items was unadjusted for potential confounders that could independently contribute to medication retraction. Further, we were unable to capture which order set items were pre-checked, which could provide further insight in assessing order set design. Our metrics derived from the EHR audit trail reflect observed clinician behaviour but are only indirect indicators of what clinicians were thinking. For example, a retracted order within 30 min may not necessarily mean that the item was initially ordered in error; there could be other reasons, such as a change in clinical condition, that warranted the order retraction. While these types of limitations are inherent in computational ethnographic studies, the broad findings from these metrics remain compelling with a large number of low bias observations captured passively from the EHR.

    Conclusion

    Real-world practice patterns extracted from the EHR audit trail demonstrate that order sets often do not align with what clinicians need at the point of care. Quantitative insights from electronic records of clinical orders can inform how clinical decision support like order sets can be optimised or replaced with other potentially more appropriate technology to facilitate clinician workflow and improve care delivery.

    References

    1. ↵
      1. Medicine NA of
      . Health IT and patient safety: building safer systems for better care. Medicine NA of, 2012.
    2. ↵
      2. Nebeker J ,
      3. Hoffman JM ,
      4. Weir CR
      . High rates of adverse drug events in a highly computerized hospital. JAMA Intern Med 2005;165.
    3. ↵
      2. Campbell JR
      . The five rights of clinical decision support: CdS tools helpful for meeting meaningful use. J Ahima 2013;84:42–7.
      OpenUrl
    4. ↵
      2. Harrison MI ,
      3. Koppel R ,
      4. Bar-Lev S
      . Unintended consequences of information technologies in health care—an interactive sociotechnical analysis. J Am Med Inform Assoc 2007;14:542–9.doi:10.1197/jamia.M2384
      OpenUrlCrossRefPubMed
    5. ↵
      2. Vaughn VM ,
      3. Linder JA
      . Thoughtless design of the electronic health record drives overuse, but purposeful design can nudge improved patient care. BMJ Qual Saf 2018;27:583–6.doi:10.1136/bmjqs-2017-007578
      OpenUrlFREE Full Text
    6. ↵
      2. Koppel R ,
      3. Cohen A ,
      4. Abaluck B
      . Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005;293:1197–203.doi:10.1001/jama.293.10.1197
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      2. Sittig DF ,
      3. Wright A ,
      4. Osheroff JA , et al
      . Grand challenges in clinical decision support. J Biomed Inform 2008;41:387–92.doi:10.1016/j.jbi.2007.09.003
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      2. Wright A ,
      3. Ai A ,
      4. Ash J , et al
      . Clinical decision support alert malfunctions: analysis and empirically derived taxonomy 2017;0:1–11.
    9. ↵
      2. Goddard K ,
      3. Roudsari A ,
      4. Wyatt JC
      . Automation bias: a systematic review of frequency, effect mediators, and mitigators. J Am Med Inform Assoc 2012;19:121–7.doi:10.1136/amiajnl-2011-000089
      OpenUrlCrossRefPubMed
    10. ↵
      2. Wright A ,
      3. Feblowitz JC ,
      4. Pang JE , et al
      . Use of order sets in inpatient computerized provider order entry systems: a comparative analysis of usage patterns at seven sites. Int J Med Inform 2012;81:733–45.doi:10.1016/j.ijmedinf.2012.04.003
      OpenUrlCrossRefPubMed
    11. ↵
      2. Starmer J ,
      3. Waitman LR
      . Orders and evidence-based order sets—Vanderbilt’s experience with CPOE ordering patterns between 2000 and 2005. AMIA Annu Symp Proc 2006;1108.
    12. ↵
      2. Jacobs BR ,
      3. Hart KW ,
      4. Rucker DW
      . Reduction in clinical variance using targeted design changes in computerized provider order entry (CPOE) order sets: impact on hospitalized children with acute asthma exacerbation. Appl Clin Inform 2012;3:52–63.doi:10.4338/ACI-2011-01-RA-0002
      OpenUrlCrossRefPubMed
    13. ↵
      2. Santolin CJ ,
      3. Boyer LS
      . Change of care for patients with acute myocardial infarctions through algorithm and standardized physician order sets. Critical Pathways in Cardiology: A Journal of Evidence-Based Medicine 2004;3:79–82.doi:10.1097/01.hpc.0000128715.42953.78
      OpenUrl
    14. ↵
      2. Fishbane S et al
      . The impact of standardized order sets and intensive clinical case management on outcomes in community-acquired pneumonia. Arch Intern Med 2007;167 http://www.ncbi.nlm.nih.gov/pubmed/17698690 doi:10.1001/archinte.167.15.1664
    15. ↵
      2. Micek ST ,
      3. Roubinian N ,
      4. Heuring T , et al
      . Before-after study of a standardized Hospital order set for the management of septic shock. Crit Care Med 2006;34:2707–13.doi:10.1097/01.CCM.0000241151.25426.D7
      OpenUrlCrossRefPubMedWeb of Science
    16. ↵
      2. Munigala S ,
      3. Jackups RR ,
      4. Poirier RF , et al
      . Impact of order set design on urine culturing practices at an academic medical centre emergency department. BMJ Qual Saf 2018;27:587–92.doi:10.1136/bmjqs-2017-006899
      OpenUrlAbstract/FREE Full Text
    17. ↵
      2. Chan J ,
      3. Shojania KG ,
      4. Easty AC , et al
      . Usability evaluation of order sets in a computerized provider order entry system. 2011:1–10.
    18. ↵
      2. Leis B ,
      3. Frost A ,
      4. Bryce R , et al
      . Standard admission order sets promote ordering of unnecessary investigations: a quasi-randomised evaluation in a simulated setting. BMJ Qual Saf 2017;26:938–40.doi:10.1136/bmjqs-2017-006898
      OpenUrlFREE Full Text
    19. ↵
      1. Centers for Medicare and Medicaid Services
      . Guidance for hospitals, Critical Access Hospitals and Ambulatory Surgical Centers Related to Various Rules Reducing Provider/Supplier Burden [Internet], 2013. Available: https://www.cms.gov/Medicare/ProviderEnrollment-and-Certification/SurveyCertificationGenInfo/Downloads/Survey-and-Cert-Letter-13-20.pdf [Accessed 2018 Oct 5].
    20. ↵
      2. Payne TH ,
      3. Hoey PJ ,
      4. Nichol P
      . Preparation and use of preconstructed orders, order sets, and order menus in a computerized provider order entry system. J Am Med Inform Assoc 2003;10:322–9.doi:10.1197/jamia.M1090
      OpenUrlCrossRefPubMed
    21. ↵
      2. Gupta A ,
      3. Das SR ,
      4. Pandey A
      . β-Blockers in myocardial infarction. JAMA 2018;319 http://jama.jamanetwork.com/article.aspx?doi=
    22. ↵
      1. Institute of Medicine
      . Best care at lower cost, 2012. Available: http://ucf-rec.org/wp-content/uploads/2012/09/IOM-Report9-6-12.pdf
    23. ↵
      2. Zheng K ,
      3. Hanauer DA ,
      4. Weibel N , et al
      . Computational ethnography: automated and Unobtrusive means for collecting data in situ for human–computer interaction evaluation studies, 2015. Available: http://link.springer.com/10.1007/978-3-319-17272-9
    24. ↵
      2. Gotz D ,
      3. Wang F ,
      4. Perer A
      . A methodology for interactive mining and visual analysis of clinical event patterns using electronic health record data. J Biomed Inform 2014;48:148–59.doi:10.1016/j.jbi.2014.01.007
      OpenUrlCrossRefPubMed
    25. ↵
      2. Zhang Y ,
      3. Padman R ,
      4. Levin JE
      . Paving the COWpath: data-driven design of pediatric order sets. J Am Med Inform Assoc 2014;21:e304–11.doi:10.1136/amiajnl-2013-002316
      OpenUrlCrossRefPubMed
    26. ↵
      2. Chen JH ,
      3. Goldstein MK ,
      4. Asch SM , et al
      . Predicting inpatient clinical order patterns with probabilistic topic models vs. conventional order sets. J Am Med Informatics Assoc 2016.
    27. ↵
      2. Zhang Y ,
      3. Trepp R ,
      4. Wang W , et al
      . Developing and maintaining clinical decision support using clinical knowledge and machine learning: the case of order sets. J Am Med Informatics Assoc 2018;0:1–5 https://academic.oup.com/jamia/advance-article/doi/
    28. ↵
      2. Koppel R ,
      3. Leonard CE ,
      4. Localio AR , et al
      . Identifying and quantifying medication errors: evaluation of rapidly discontinued medication orders submitted to a computerized physician order entry system. J Am Med Inform Assoc 2008;15:461–5.doi:10.1197/jamia.M2549
      OpenUrlCrossRefPubMed
    29. ↵
      2. Adelman JS ,
      3. Kalkut GE ,
      4. Schechter CB , et al
      . Understanding and preventing wrong-patient electronic orders: a randomized controlled trial. J Am Med Inform Assoc 2013;20:305–10.doi:10.1136/amiajnl-2012-001055
      OpenUrlCrossRefPubMed
    30. ↵
      2. Hickman T-TT ,
      3. Quist AJL ,
      4. Salazar A , et al
      . Outpatient CPOE orders discontinued due to ‘erroneous entry’: prospective survey of prescribers’ explanations for errors. BMJ Qual Saf 2018;27:293–8.doi:10.1136/bmjqs-2017-006597
      OpenUrlAbstract/FREE Full Text
    31. ↵
      2. Hulse NC ,
      3. Lee J
      . Extracting actionable recommendations for modifying enterprise order set templates from CPOE utilization patterns intermountain healthcare, Salt Lake City, UT; 2 Department of Biomedical Informatics. AMIA Annu Symp Proc 2017:950–8.
    32. ↵
      1. Institute for Safe Medication Practices
      . ISMP’s guidelines for standard order sets 2010:1–5.
    33. ↵
      2. Lowe HJ ,
      3. Ferris TA ,
      4. Hernandez PM , et al
      . STRIDE—an integrated standards-based translational research informatics platform. AMIA Annu Symp Proc 2009;2009:391–5.
      OpenUrlPubMed
    34. ↵
      2. Klann JG ,
      3. Szolovits P ,
      4. Downs SM , et al
      . Decision support from local data: creating adaptive order menus from past clinician behavior. J Biomed Inform 2014;48:84–93.doi:10.1016/j.jbi.2013.12.005
      OpenUrlCrossRefPubMed
    35. ↵
      2. Weisstein EW
      . Bonferroni correction. Available: http://mathworld.wolfram.com/BonferroniCorrection.html [Accessed 11 Oct 2018].
    36. ↵
      2. Yancy CW ,
      3. Jessup M ,
      4. Chair V , et al
      . ACCF/AHA guideline for the management of heart failure: executive summary: a report of the American College of Cardiology Foundation/American Heart Association Task Force on practice guidelines. JAC 2013;62:1495–539.
      OpenUrl

    Footnotes

    • Contributors RCL led the conception and execution of the project, including writing of the manuscript. JKW assisted in the analysis. CS assisted in the review of the manuscript. JHC supervised the study as the PI.

    • Competing interests None declared.

    • Patient consent for publication Not required.

    • Ethics approval The study was approved by the institutional review board.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data availability statement Data are available on reasonable request.