Improving emergency department patient-doctor conversation through an artificial intelligence symptom taking tool: an action-oriented design pilot study

The aim of this study

In this study we evaluated a prototype digital history/symptom taking and handover system, which included a patient-facing tool for symptom and history taking. Through the patient’s interaction with this, an HCP clinical handover report was generated and displayed to the physician and/or nurse, via an HCP-facing tool. Our hypothesis was that some of the rapport, communication, documentation and time pressures in the crowded ED could be alleviated by the use by patients of a modified digital symptom assessment tool in the waiting room with transfer (handover) of a symptom assessment report to the HCPs in the ED. This pilot study was based on the approach of Furaijat et al.¹⁰, in which the quality improvement delivered by a prototype patient facing system was assessed in two study stages. Phase I, the action-oriented phase, included the initial implementation of the patient- and HCP-tools (version V1), and their evaluation by all users. The feedback on performance from Phase I was used to create a modified system (version V2), which was then further evaluated in Phase II. Patients, physicians and nurses quantitatively evaluated the two system versions in terms of their usability and usefulness in facilitating patient-HCP conversation and rapport formation in the ED setting. For HCP-users, we also explored the helpfulness of the medical information provided at handover and the potential of the system to save HCP time.

Design: overall study approach and study type

Phase I followed an action-oriented approach ¹⁰ which involved close collaboration between researchers and HCPs. Modifications suggested by users that could be implemented within the project timeframe were then used to develop a V2 prototype, and at a switch-over point, the V1 tool was replaced by the V2 tool, which was then fixed and remained unchanged during Phase II. There were no study design changes made between Phase I and II.

This study explored the potential for ED service quality improvement through the introduction of an augmentative prototype history and symptom taking tool. Questionnaire-based user perceptions on the potential for the system were collected alongside qualitative observations. No interventional patient outcomes measures were recorded. The system was not closely integrated into clinical workflows during the study, and it did not replace standard practice in the ED. HCPs were trained not to rely on the prototype system for formal or definitive symptom-taking support during the study. There was nonetheless a theoretical risk to patients from the study of either inappropriate prolongation of the waiting time or through inappropriate reliance of HCPs on the system. For this reason, a designated physician (JSB) made a safety analysis for each patient by examining the handover report and the patient’s clinical history in the electronic health record and adjudicated if there was any evidence of inappropriate physician guidance. This study was conducted in accordance with requirements of the SQUIRE ¹¹, SPIRIT-AI and eCONSORT ¹² guidelines, as described in the checklist (Supplementary Table 1).

Description of the prototype digital symptom/history-taking system

The system consisted of patient- and HCP-facing tools (Figure 1). The patient-facing tool was provided to recruited patients in the waiting room on a tablet computer (iPad→, Apple Inc.). The patient history taking tool used a cloud-based probabilistic reasoning engine combined with curated medical knowledge to ask the patient the optimal set of questions based on probable conditions and condition urgency. The Ada reasoning engine versions used in the study were 1.31.1 and 1.31.2, between August 3^rd - 21^st 2020. The assessment was performed as a question flow with options to confirm, deny or skip the question. The tool asked successively tailored questions about the respondent’s medical history, the main presenting complaint(s) and related attributes of their symptoms such as severity and time course.

• Download figure
• Open in new tab

Figure 1.

The prototype digital history/symptom-taking and handover system evaluated in this study, showing the interactions between the patient-facing and HCP-facing tools and describing how artificial intelligence reasoning engine functions to ask the patient, sequentially, the most relevant next question.

The HCP-facing tool provided a secure web-interface dashboard that listed all completed assessments to the ED clinical staff. The tool also provided a detailed handover report, designed to provide clinical information quickly and safely to HCPs. The handover report included the patient’s basic information (sex, year of birth), basic medical history information (smoker, hypertension, diabetes and pregnancy status), main presenting symptom(s), details of these symptoms, including the specific questions asked by the app, and answers provided by the patient.

Design: study procedure

The procedure of the study is described in Figure 2.

• Download figure
• Open in new tab

Figure 2.

Overview of the study procedure.

Study measurements

After examination by the ED physician, the patient and the ED physician completed separate questionnaires, with evaluation ratings of the tool on modified Likert scales. Nurses completed the same questionnaire as doctors when possible, however, it was recognized at the time of trial design, that there would not always be a nurse-patient interaction after triage in which the handover report is relevant.

Study setting

The study was conducted in the ED of the Katharinenhospital Stuttgart which is an adult tertiary referral and major trauma hospital in South-Western Germany It provides interdisciplinary emergency treatment for 100-120 patients per day. The center adopts the First View Concept ¹⁴, in which an emergency registrar/consultant sees each patient in an interdisciplinary approach. The center has 23 treatment rooms with central monitoring, one resuscitation rooms, one wound and one plaster room. It uses the internationally recognized Emergency Severity Index (ESI) triage system to guide the treatment of emergency patients according to medical urgency ¹³.

Sample size calculation

This study was designed for real-world tool optimization (in the action-oriented Phase I), followed by a preliminary observation assessment of the tool’s potential in the ED. It was also designed as guide to a later larger trial. The study sample size was determined by the requirement to evaluate the study safety criterion of identifying any adverse events of 5% using the approach of Viechtbauer et al.¹⁵: if a safety problem that exists with 5% population probability then the problem would be identified with 95% confidence in a sample size of 59 participants. The sample size calculation was for all patients recruited in both study phases, on the basis that no changes to the core AI assessment or core information in the handover report between versions would be carried out.

Data analysis

The quantitative data were analyzed using standard Python (version 3.7.4) statistical modules (scipy module version 1.3.0) using descriptive statistics and the Mann-Whitney U test, a nonparametric test of statistical significance suitable for categorical data ¹⁶. For statistical significance testing, the value of α (here α = 0.05) was adjusted using the Bonferroni correction, α/m, where m is the number of questions evaluated for each group. For patients m = 6 and α_corrected = 0.0083 patients, for HCPs m = 4 and α_corrected = 0.0125 and for the degree of patient self-sufficiency m = 1 and α_corrected = 0.05.

Recruitment

The total number of recruited patients was 81 (41 women, 40 men) with 45 patients in Phase I and 36 in Phase II (Figure 3). Of the 81 recruited patients there were three ESI level 3 patients, 77 ESI level 4 patients and one ESI level 5 patient. A detailed description of the patient population is provided in Table 1 and the full results for each patient are in Supplementary Table 2.

• Download figure
• Open in new tab

Figure 3.

Participant Recruitment Flow

Baseline Data

Patient populations in the two study phases were similar (Figure 3 and Table 1): the mean (SD) age was 38.7±15.0 years for all patients, 36.0±15.9 for Phase I patients, and 42.1±13.1 for Phase II patients. The dominant medical classifications (by ED discharge diagnosis) were Orthopedics; Internal Medicine; Neurology and Surgery.

Missing Data

Questionnaire completion rate was 100.0% for patients (of which 90.1% completed all questions), 96.3% for physicians and 12.3% for nurses (Figure 3, Table 1 & 2). Nurse questionnaires were only completed when they took part in with symptom/history information, and when ED busyness allowed. For all survey questions, the analysis approach was to report all data with respect to the number of responses to that survey question (i.e. with the denominator in analyses being lower where there was missing data).

Safety Endpoint

A safety analysis was carried out for each patient by examining the handover report and the patient’s clinical history in the electronic health record, which were considered together, and alongside feedback from the treating HCPs, to determine if there was any evidence of inappropriate HCP guidance. The conclusion of this evaluation was that there were no cases of HCPs being guided in an unsafe manner by the handover report for the 81 recruited patients.

Summary of changes made between the V1 and V2 systems

Three changes were made in total:

1 - The addition of three free-text fields, where the patient can supply initial information on their medical history, current medications, and allergies.

This change was requested by ED-physicians so that patients would have the opportunity to pass on information, in their own words, that was not always collected by the tool’s question flow. This was implemented in a manner that did not change the core AI-based symptom assessment. This change affected the information entered into the patient-facing tool, and the information presented on the HCP-facing tool.

2 - Improve the user interface at the transition between patient information and symptom assessment

A minor rephrasing was made to the patient-facing tool in response to a small number of patients who reported being confused by the initial wording.

3 - Fix of a minor bug affecting the HCP-handover report

A minor bug was removed which had resulted in a small number of handover reports not accurately displaying the transcript of questions that Ada asked the patient alongside the patient response.

The changes were anticipated to have a minor impact on the evaluation of the tool. The patients’ ratings of the tool (Table 2) improved by 9.5% and the physicians by 9.1% across all ratings with a significant improvement (p = .003, α_corrected = 0.0083) in the responses to the patient question ‘(i) was the use of the tool and the answering of its questions interesting for you?’. The improvements in nurse ratings of the system between Phase I and II are reported in Table 2, however, there were insufficient nurse assessments in Phase I to allow statistical significance testing.

Patient, physician and nurse ratings of the tool

Patients were positive or highly positive (90.1%) about how interesting the tool was to use, its understandability (86.4%), its usability (83.5%), its ability to facilitate understanding and rapport with the HCPs (75.3%) and about recommending the tool to peers (83.5%) (Table 2). Likewise, 73.1% of physicians were positive or highly positive about the tool’s potential for facilitating understanding and rapport with patients.

Nurses were even more positive or highly positive about the tool’s potential for facilitating understanding and rapport with patients (100.0%), about the helpfulness of the medical information provided (100.0%), about the tool’s time saving potential (80.0%) and about recommending the tool to peers (100.0%).

Subanalyses by medical specialism

A subanalysis by medical specialism of ED discharge diagnosis is shown in Table 3 and described in detail in the Supplementary Information. This subanalysis showed there was lower understandability of the tool by Neurology patients, and these patients also gave lower ratings to the conversation facilitation provided by the tool. Physicians rated the level of helpful information, the time saving utility of the report and their likelihood to recommend the system more positively for Internal Medicine handovers than for handovers for all recruited patients or for other specialisms.

Degree of patient assistance provided

Across both phases, most patients were able to use the tool with little or no assistance (76.3%). This measure improved significantly between Phase I (63.6%_ and II (91.7%, p = .003, α_corrected = 0.05).

Variability of physicians’ perceptions

Patient symptom and history data handover was evaluated for 96.3% (78/81) of the patients by 24 different ED physicians. Qualitative interviews with physicians in Phase I of the study revealed a number of physicians who were exceptionally enthusiastic about the performance of the system, i.e. an ‘early adopter’ mentality. The quantitative analysis of the distribution of Likert scores (see Supplementary Figure 1) supported these qualitative findings, in that there is a skewed distribution of physician scores, with two physicians (one of whom evaluated handover reports for 6 patients), providing highly positive evaluations of the tool.

Overall results discussion

This study used methodologies that have been adopted in other ED quality improvement evaluations of new digital technologies^10,17, to evaluate a novel AI-based symptom/history taking and handover system, by means of patient and HCP questionnaires. The performance of the system was evaluated in two principle areas: (i) the facilitation of bidirectional patient-to-clinician conversation and rapport building; and, (ii) the potential to save HCP time through symptom collection and documentation facilitation.

There was a strongly positive rating by patients, physicians and nurses of the tool as an aid in patient-to-clinician conversation, communication and rapport building. The proportion of physicians positive/highly positive about recommending the system to their peers (53.2%) was not as high as it was for patients and nurses (77.9% and 100.0% respectively). Overall, physicians had mixed views on the degree of helpful clinical information and time saving potential of the clinical handover report. Underlying the mixed results in the physician ratings (Supplementary Table 2), were a large number of patients for whom the ED physicians described the presence of a ‘visual diagnosis’ - i.e. one that was immediately apparent by simply looking at a patient, and for whom patient-directed symptom taking, in whatever form it is designed, is unlikely to save history-taking and recording time. For some other patients, the tool did not adequately draw together and summarize a highly complex, sometimes multimorbid medical presentation in a manner that made it useful for physicians. Both qualitative findings and quantitative analysis revealed a subset of highly enthusiastic early-adopter physicians, as have been recognized in other studies ¹⁸, who were highly positive about the history and symptom taking information provided by the tool and for its potential to improve rapport, to provide helpful clinical information and to save ED time.

The relatively lower understandability of the tool by Neurology patients, and the lower rating of its conversation facilitation, may reflect effects of neurological symptoms on their use of the tool. Our interpretation is that the positive/highly positive evaluations from physicians for Internal Medicine handovers is likely due to the system’s AI reasoning engine being better at directing the question flow for conditions which have many subtle interlinked clinical symptoms.

One aim for the development of a waiting room patient history taking - and HCP handover tool is that it be easy to use without assistance. Many patients visit only once, so they would have little opportunity to learn to use the tool over time. In this study, patient assistance was provided where needed by the recruiting physician (JSB) to maximize data collection for the action-orientated tool improvement and so that the ED-physician/nurse facing interface’s performance could be assessed with meaningful clinical information. Most (76.3%) patients required little or no help with the use of the tool, and in some subspecialisms, patients were highly independent using the tool (93.8% in Internal Medicine with no subspecialism), whereas 66.7% of Orthopedics patients required little or no help. A priority in the future development of the prototype will be to adapt the reasoning engine and the user interface to minimize the level of help required for all patient groups, and this may involve making the tool available in the patient’s primary language instead of in the national language only.

The action-orientated approach with two study phases provided an opportunity within a clinical trial framework to rapidly modify a prototype tool’s design, based on feedback from patient and physician evaluations, and to further evaluate the modified tool. Comparison of tool ratings showed statistically significant improvements in performance between the V1 and V2 tools in two evaluation categories: (i) how engaging/interesting patients found the tool to use; and, (ii) the degree of self-sufficiency of the patients in using the tool. These improvements can be explained through improvements made in tool usability and in functionality added to the tool to capture additional information about their symptoms and history. Many useful insights on tool performance and usability were obtained in the study. Only a relatively small number of optimizations could be executed between the V1 and V2 tools, due to the need to prioritize in-study changes to those that could be completed within the possible study recruitment period. The remaining insights, including those on the V2 tool will be used in later prototype development and optimization.

There are no previous studies describing similar digital history/symptom-taking and handover systems in the literature. Arora et al. (2014) ¹⁹ explored patient impressions and satisfaction of a self-administered, automated medical history-taking device in the ED, and they also reported high levels of patient enthusiasm in the evaluated tool and for the potential for rapport-building with HCPs. However, handover to HCPs was not carried out in their study, nor did the tool have a system to enable handover. A number of studies ^9,20,21 have reported apps for self-assessment of urgency/triage in the general practice and acute primary care setting, but evaluation of the handover of history and symptom information to HCPs was not addressed.

Clinical and Policy Implications and Implications for Future Research

It is recognized that empathy and rapport can be lacking in EDs, both of which are important for staff stress levels and satisfaction and for patient outcomes and empowerment ^1,5,6,22,23. Although a digital tool can never be a complete solution to improve human-to-human empathy, this study has nonetheless shown patient and HCP enthusiasm for the rapport building and conversation facilitation enabled through such tools. This study evaluated a prototype, and optimizations of this tool are required, based on the insights from this study, to facilitate patient-HCP conversations and optimize rapport in a manner which is complete (i.e. clinically helpful information for every patient), streamlined and seamless (i.e. no additional tasks or systems for clinicians). This will then be followed by clinical trial evaluation of a completed tool.

Although the results of this study were definitive on the potential for rapport and conversation facilitation in the ED, they were equivocal on the potential for the tool, as it is currently implemented, to save clinician time, and clinicians could identify time saving only in some types of patient presentations (Internal Medicine, Surgery). Nurses were more positive about the general potential of the tool for clinician time saving. It is recognized that further development of the prototype is required if the aim of the completed approved regulatory product is to deliver definitive physician time savings for all patients. The role of a patient symptom and history taking tool, is not only potentially important in saving clinician time, but also in contributing to documentation accuracy and completeness, as it is known that medical performance reduces with stress and over-stretching in the ED, and is likely to result in greater error making, including in documentation ^4,5. It was not possible to measure the contribution of the tool to documentation completeness and accuracy in this study.

Limitations

This was a single site study in Germany and the findings may not be generalizable to other facilities or to other countries. The patient and physician evaluations were not influenced by selection bias, however, as the nurse evaluations were completed on a voluntary basis the influence of selection bias cannot be excluded and 90% of the nurse evaluations relate to study Phase II. This study explored tool use in the German language and in the German setting only, and the sample size was relatively small sample given resource constraints within the ED. The study did not include pediatric patients. Infection control with patients using a tablet in the ED could be challenging. This study explored the early phase after prototype system implementation, training and first experience of its application by users. Although minimally relevant for ED patients, the skill and speed of ED physician and nurse use of the new tool, and their perception of its performance is likely to change over time. Despite limitations, our approach allowed us to investigate implementation at this single site in considerable depth.