Article Text
PDF
Abstract
Health workers in low-resource settings often lack the support and tools to follow evidence-based clinical recommendations for diagnosing, treating and managing sick patients. Digital technologies, by combining patient health information and point-of-care diagnostics with evidence-based clinical protocols, can help improve the quality of care and the rational use of resources, and save patient lives. A growing number of electronic clinical decision support algorithms (CDSAs) on mobile devices are being developed and piloted without evidence of safety or impact. Here, we present a target product profile (TPP) for CDSAs aimed at guiding preventive or curative consultations in low-resource settings. This document will help align developer and implementer processes and product specifications with the needs of end users, in terms of quality, safety, performance and operational functionality. To identify the characteristics of CDSAs, a multidisciplinary group of experts (academia, industry and policy makers) with expertise in diagnostic and CDSA development and implementation in low-income and middle-income countries were convened to discuss a draft TPP. The TPP was finalised through a Delphi process to facilitate consensus building. An agreement greater than 75% was reached for all 40 TPP characteristics. In general, experts were in overwhelming agreement that, given that CDSAs provide patient management recommendations, the underlying clinical algorithms should be human-interpretable and evidence-based. Whenever possible, the algorithm’s patient management output should take into account pretest disease probabilities and likelihood ratios of clinical and diagnostic predictors. In addition, validation processes should at a minimum show that CDSAs are implementing faithfully the evidence they are based on, and ideally the impact on patient health outcomes. In terms of operational needs, CDSAs should be designed to fit within clinic workflows and function in connectivity-challenged and high-volume settings. Data collected through the tool should conform to local patient privacy regulations and international data standards.
- diagnostics and tools
- child health
- public health
- treatment
- health systems
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Key questions
What is already known?
Clinical decision support tools deliver evidence-based recommendations to support clinical decision-making such as prescribing antibiotic treatment.
Although widely integrated in many healthcare activities in high-income countries, in low-resource settings handheld-based clinical decision support algorithms (CDSAs) are being developed and have shown to provide benefits to patients (outcome) as well as to the public (reduction of antibiotic overprescription).
What are the new findings?
To support end user curative and preventive consultations at point of care in low-resource settings, stakeholders defined CDSAs that are based on evidence, human-interpretable and customisable to the setting and its workflow, and importantly incorporate contextual data and diagnostic test performance in the patient management recommendation delivered.
What do the new findings imply?
CDSAs can help deliver quality health services by providing evidence-based recommendations during patient consultations.
The use of such tools, in combination with available resources, aims to improve patient outcome and rational use of diagnostics and therapeutics at point of care, and in the long term reduce economic hardship on patients and health systems.
Introduction
Health workers at primary healthcare (PHC) have the greatest challenge of ensuring appropriate care for their communities. One way to support health workers is through the provision of clinical guidance in the form of clinical practice guidelines or clinical decision support algorithms (CDSAs). Clinical practice guidelines are a set of recommendations on diagnostic and treatment modalities based on systematic review of evidence and assessment of the benefits and harms to patients. They aim to deliver the best care and rational clinical decisions to improve patient outcome. The WHO developed several simple clinical decision guidelines to support health workers in low-income and middle-income countries (LMICs) in the evaluation and management of clinical problems, with the ultimate goal of improving quality of care.1 These include the Integrated Management of Childhood Illness (IMCI) designed in the 1990s for common diseases affecting children younger than 5 years of age,2 which was later adapted for use in the community as the integrated Community Case Management guideline.3 Other guidelines have been developed and either take an integrated management approach similar to IMCI, such as the Integrated Management of Adolescent and Adult Illness,4 antenatal care (ANC) and family planning, or a disease-specific approach (ie, WHO’s guidelines for diagnosis, treatment, prevention and control of dengue5).
While the global health community awaits updated and more comprehensive clinical algorithms, the fourth industrial revolution6 is seeing developing economies ‘leapfrogging’7 mobile technologies across multiple sectors, including in health. As a result, a plethora of digital clinical decision support systems (CDSS) have been implemented in LMICs. CDSS is defined as ‘digitized job aids that combine an individual’s health information with the health worker’s knowledge and clinical protocols to assist health workers in making diagnosis and treatment decisions’.8 It is well recognised that adherence to standardised clinical practice guidelines is a measurement of quality of care,9 10 and that improving adherence to evidence-based guidelines is associated with better health outcomes.11 By digitalising clinical guidelines into electronic CDSAs, in its simplest form a decision tree, several groups have shown significant increase in health worker adherence to guidelines compared with routine care (ie, CDSA vs paper-based IMCI).12–14
However, today CDSAs carry limitations similar to the guidelines and still lack point-of-care (POC) data that could improve the accuracy of diagnoses and treatment recommendations. In terms of clinical guidelines related to febrile illness, the only diagnostic test widely used is a malaria rapid diagnostic test (mRDT), and while this inclusion was important in the context of ‘test-and-treat’15 it is limited to ruling in or ruling out one pathogen alone. Acute febrile illness can be caused by bacterial or viral respiratory infections, or seasonal, geographically localised pathogens like dengue. The inclusion of additional simple wet and digital diagnostics into adapted algorithms can reduce antimicrobial overprescriptions as well as improve the child’s health outcome.16 17 This was demonstrated with IMCI-based CDSAs, and it is postulated that similar benefits on outcome and antimicrobial use could be awarded to other use cases.
With the growing development of CDSAs, in particular those based on widely adapted guidelines that have the potential to reach thousands to millions of people, including vulnerable groups, it is important to ensure that new tools meet the needs of end users. Establishing a target product profile (TPP) for such a toolkit is an important step towards meeting this goal. TPPs include minimal and optimal definitions of the characteristics of a proposed health tool to address a defined public health need. Ideally the tool is designed to meet the majority of optimal characteristic criteria as feasible, while still meeting the minimal criteria.
To this end, here we present a TPP that was defined through consensus with stakeholders and experts. It aims to inform developers and implementers alike of the requirements for CDSAs aimed at guiding health workers during preventive or curative consultations with any type of patient (eg, children, adolescents, adults, pregnant women and so on) using, whenever feasible, available diagnostic tools.
Methods
Draft TPP
A draft TPP (version 0) for CDSAs combined with POC results was developed (by KGP, SD, RS, ZK, GLM, FGM) based on standard procedures at the Foundation for Innovative New Diagnostics (FIND) and WHO, where characteristics are defined as ‘minimal’ and ‘optimal’ criteria (online supplementary file 1). ‘Minimal’ is used to refer to the lowest acceptable output for a characteristic and ‘optimal’ for the ideal target of a characteristic. The comment sections in a TPP are used to explain or provide examples of the characteristics. As this TPP was for a multicomponent toolkit, CDSA and POCs, the structure of the draft TPP was adapted to cover the different components and for the purpose of clarity. These sections included general scope, scope of toolkit components, electronic clinical decision support algorithm, POC tools, app, data and procurement, and were also the sections used in the final TPP.
Supplemental material
Expert meeting
To discuss selected aspects of the draft TPP (algorithm validation, performance, machine learning (ML), diagnostic data, disease prediction, clinical workflow and application functionality) (online supplementary file 1), a meeting on this subject was convened by FIND and WHO involving 39 experts from academic institutions, industry, and private and public sectors from 11 countries.18 Experts were selected based on their experience in relevant areas of work directly related to digital health (clinical decision support or other digital tools targeting resource-challenged settings) and/or diagnostics (wet and digital). Following the expert meeting, the TPP was updated based on expert feedback (version 1.0) (online supplementary file 2).
Supplemental material
Delphi process
To facilitate consensus building for the TPP, we followed a Delphi-like process where TPP version 1.0 was reviewed through a three-part online survey. The first part of the survey collected professional information from the survey respondents, as well as experience with clinical practice/research, diagnostic test development, software development, implementation of healthcare programmes in LMICs and health information systems. The second part of the survey allowed for rating of both minimal and optimal criteria for all TPP characteristics. Agreement was scored on a Likert scale from 1 to 5, where 1 corresponds to fully disagree, 2 mostly disagree, 3 neither agree nor disagree, 4 mostly agree and 5 fully agree. Disagreement with a criterion was based on a rating from 1 to 3 and required a comment/suggestion by the survey respondent. To reach consensus, more than 75% of survey respondents should provide a rating of at least 4 (agreement) on the Likert scale. The third part of the survey provided an opportunity for respondents to share comments/feedback/ideas on the TPP if not covered in the characteristics section. After the first round of the survey, agreement for each characteristic was calculated as a percentage, and respondent comments, whether associated with agreement or disagreement, were reviewed. The TPP was revised based on reviewer feedback and sent for a second survey. The data were compiled as in the first round. If 75% agreement was achieved for all characteristics, the Delphi process was closed.
Patient and public involvement statement
At no point in this work were patients or the public involved in the design of this study or in the reporting and dissemination of the results.
Results
Expert consensus on the TPP through a Delphi process
In the first round, the online survey was sent to 77 experts (including workshop participants), and 28 responded to the survey (response rate, 36.4%), with some respondents submitting a coordinated response for their institution. Twenty-five per cent of the respondents were from LMICs (7 of 28); however, the majority of the respondents (22 of 28, 78%) had experience working in Africa and 57% (16 of 28) in South-East Asia. More than 85% (24 of 28) of the respondents had more than 5 years of experience in clinical practice and research, 50% (14 of 28) in health programme implementation in LMICs, 39% (11 of 28) in health information systems, 21% (6 of 28) in software development and 14% (4 of 28) in diagnostic product development (online supplementary file 3). Most participants work in international organisations (7 of 28, 25%) and academic institutions (10 of 28, 35.7%). In the first round, at least 50% agreement was reached for all criteria of all the characteristics, and feedback was taken into consideration to develop TPP version 1.1 for the second round of the Delphi survey (online supplementary file 4). In this new version, the characteristic ‘Encounter’ was removed as several participants expressed it was redundant with the characteristic ‘Workflow’. Therefore version 1.1 had 40 characteristics.
Supplemental material
Supplemental material
A second online survey was sent to the 28 respondents from round 1. In this round, 23 participants responded to the survey (response rate, 82%). Most participants were from academia (7 of 23, 30%), non-profit international organisations (7 of 23, 30%) and had a similar profile to respondents from round 1 (online supplementary file 3). In this round of review, at least 75% of the participants agreed with the minimal and optimal criteria for all TPP characteristics. On average, 91% of survey respondents agreed with the minimal and optimal criteria (figure 1) and the TPP was finalised (version 2).
Figure 1Expert consensus on TPP characteristics. Expert agreement is shown for each TPP characteristic minimal (Min) and optimal (Opt) criteria. Agreement was scored on a Likert scale from 1 to 5, 1 = fully disagree, 2 =mostly disagree, 3 =neither agree nor disagree, 4 =mostly agree and 5 =fully agree. Consensus was reached when more than 75% of survey respondents provided a rating of at least 4.CDS, clinical decision support; POC, point-of-care; API, application programming interface
Final TPP
General scope and Scope of toolkit components
This TPP defines a toolkit composed of CDSAs and POCs intended to support evidence-based clinical decisions made by health workers during a preventive or curative consultation, by capturing patient clinical and context-specific data, as well as diagnostic test results, to provide recommendations on diagnosis and patient management (including counselling, follow-up visits and so on). The clinical algorithm integrates POC results and is embedded in an app. POCs in this toolkit should be regulatory approved for use in the setting of implementation.
The experts agreed that the CDSA should first define its target: (1) patients (eg, age group), (2) medical problems (eg, syndrome or disease), (3) end users (eg, minimally trained health worker or trained clinician) and (4) level of care (eg, community or health facility) for which the clinical recommendations are meant and relevant. These specifications should be clearly communicated to the end user at the beginning of the algorithm. The health worker should immediately be warned if the information she or he enters in the system does not fit with these specifications. For example, if the scope of a CDSA is limited to patients aged 6 months to 5 years and the user enters 2 months of age, the CDSA logic should prompt an incongruence of the data entry and this should be appropriately reported back to the user. In cases where an app proposes different algorithms for different use cases (ie, ANC, management of childhood illness), the specifications shall be clearly available for each individual algorithm. For example, if a CDSA allows for consultations related to ANC for pregnant women as well as for neonates, the algorithm should request specific patient information to triage patient along different algorithm branches respective to the use case (table 1).
Minimal and optimal target product profile characteristics focused on the general scope of the toolkit and toolkit component characteristics, as defined by expert consensus process
Clinical decision support algorithms
Regarding the algorithm’s medical content, the working group agreed that it must be evidence-based. Whether newly developed or adapted from existing guidelines, the algorithm’s medical content should be developed following international standards for the development or adaptation of evidence-based clinical guidelines, to ensure it implements the best available evidence. This evidence can originate from well-established clinical guidelines such as WHO/international/local clinical care guidelines, or from peer-reviewed articles, clinical practice and/or validation research, and the level of evidence and the strength of recommendations in the target context should ideally be communicated. In line with this, recommendations should cause the least harm to the patient and to the community. For example, algorithms related to treatment and management of a patient suffering from infectious diseases should also support antimicrobial stewardship to avoid unnecessary side effects for the patient and the development of resistance in the community and at the global level. In addition, the algorithm should be validated both analytically and semantically to ensure that the algorithm output is accurate and reproducible, does not deviate from expert content/evidence, and that there are no interactions or conflicts in the logic.
Furthermore, important characteristics were content quality, ML. With the rise of ML also in this domain of CDSAs to analyse the data captured through algorithms and then for example augment algorithm diagnostic or prognostic accuracy, workshop experts thought it important to address this in the TPP. It was agreed that an algorithm that recommends treatment decisions and management of patients should be at a minimum human-interpretable, meaning one should understand how input data are processed into output data. Changes to the algorithm logic based on ML analyses should also be validated (table 2).
Minimal and optimal target product profile characteristics focused on the electronic clinical decision support algorithm and POC tools, as defined by expert consensus process
POC tools
POC tools are defined here as medical diagnostic resources that can be used by health workers immediately at point of care to measure clinical parameters (ie, vital signs, host biomarkers and pathogen biomarkers) or visualise a patient’s tissue. They vary in design complexity (lateral flow vs table-top instrument), reagent, ease of use and cost. These tools are often budgeted and resourced by ministries of health but may not always be available in facilities, in particular at lower levels of the healthcare system. Using these tools during patient consultations and incorporating their data into the algorithms are dependent on their availability and whether the CDSA was designed to compute those data into triage or recommendation outputs. Experts agreed that the post-test probability or the likelihood of a disease being present should be assessed based on the pretest probability—that is, the prevalence of the disease in the corresponding patient population; and the diagnostic positive/negative likelihood ratio—that is, how good a test predicts the presence or absence of disease, of all clinical information (ie, signs, symptoms and POC results) (table 2). Clinical diagnosis often lacks accuracy, and therefore diagnostic tests that have the proven added value for patient management should be integrated within algorithms if the setting allows for it. These tests can be wet diagnostics such as pathogen-specific rapid diagnostic tests or handheld devices such as haemoglobinometers, and do not exclude digital diagnostics such as digital thermometers.
App
An important characteristic in this section is system validation. Today, there is no standard process to validate and assess the performance accuracy of CDSAs. There is also a severe lack of evidence and peer-reviewed publications on both the development and the performance of these tools in LMICs. While some groups have performed randomised controlled trials (RCTs) to measure patient outcome and other parameters that impact patient health, such as the reduction of antibiotic overprescription,16 17 others have assessed the performance of their tool by measuring health worker adherence to guidelines.12–14 The level of validation ultimately depends on the type of algorithm. The US Food and Drug Administration has provided some guidance on the kind of evidence that needs to be provided.19 The TPP criteria are inspired by this guidance. The minimal criteria guarantee that the app has a CDSA that is based on evidence (clinical association) and that input data are processed correctly (analytical validation). The optimal criteria include clinical validation, in addition to clinical association and analytical validation. Through clinical validation, the app has been assessed for the intended outcome (ie, improved clinical outcome, better quality of care, rational use of resources, reduction of antibiotic overprescription and so on) (table 3).
Minimal and optimal target product profile characteristics focused on app characteristics, as defined by expert consensus process
In May 2017, a new regulation (EU 2017/745) on medical devices, to which CDSA pertains by definition,20 was adopted by the European Commission, which will be applied starting Spring 2020.21 This regulation includes a reinforcement of the rules on the clinical evidence that should be provided by developers and manufacturers, at least if based in Europe. While waiting for LMICs to have their own regulations, this new regulation will certainly impact development and validation by European developers.
Experts also called for application programming interface to allow other software to ‘talk to’ and interact with the tool through commonly known programming languages. The app should be protected by authorisation and authentication that, as a minimum requirement, adhere to usual technical standards, and optimally would adhere to the HIE (Health Information Exchange) and/or HL7 (Health Level Seven) international22 standards. It was noted that adherence to HL7 can be expensive; hence, this was proposed as an optimal rather than a minimal requirement. The app should also be designed to integrate with patient care pathways and health worker workflows with limited disruption. The app accordingly has pause and resume capability to allow time for diagnostic tests to be performed and the results entered in the algorithm. POCs such as mRDTs can require up to 20 min for results to be read, depending on the manufacturer.
Data
Digital tools have the potential to amass enormous amount of data. With CDSA, patient personal data are captured, stored and potentially used for other operational (ie, supply chain management) and algorithm development purposes. Many countries have personal data privacy and protection laws in place to protect citizens from the abuse of their personal information. In the European Union this is known as the General Data Protection Regulation (GDPR).23 However, there are still many countries around the world that do not have a national eHealth strategy or personal data legislation. This called for the need to address personal data handling with CDSA for security and accountability purposes. Experts agreed that the implementer, who would be considered the data controller, is responsible for the data and should comply with local data policies and legislations. No specific legislation was mentioned as this differs by country and region. In addition, the history of the processes that are applied to data as well as the origin of data should be documented (data provenance). The app should function under secure connectivity to avoid loss and corruption of sensitive data, and mitigate cyberattacks, whether data are at rest or in transmission (table 4).
Minimal and optimal target product profile characteristics focused on data characteristics, as defined by expert consensus process
Discussion
In this work, our aim was to develop a TPP to enable the development of a toolkit that includes CDSAs that are based on evidence and designed to integrate the results of the diagnostics performed at point of care to guide rational clinical decisions. The use of such toolkit would aid in strengthening healthcare particularly at the community and primary care echelons, and help address health worker difficulties in differentiating diseases and syndromes without diagnostic tests and the aid of formalised diagnostic processes, which can be provided through algorithms.
In terms of TPP characteristics, ‘Data’ and ‘Validation’ emerged as key topics of discussion. Data generated from CDSA require storage and connectivity and are therefore vulnerable to abuse, loss or unlawful purposes. Although many countries have enacted personal data protection legislation, many fall short on enforcing these and others are still drafting bills. For example, in the African region, in 2017, 17 (out of 54) African countries had put in place comprehensive personal data protection legislation.24 Several regimes in the region are starting to draft legislation on processing and movement of personal data25; however, funding attrition and the lack of regulators cripple the process. That said, there are continent-wide initiatives that promise to set data protection framework, such as the EU’s GDPR. The African Network of Personal Data Protection Authorities, a cooperation of eight African countries, seeks to draft data protection laws, formulate opinions on specific issues and establish a consultative framework on data protection. There are also regional conventions for data protection and privacy that have been enacted, such as the Southern African Development Community Model Law26 for the south and the Act on Personal Data Protection within the Economic Community of West African States for the west.27 Unlike their southern and western counterparts, east African nations have not adopted similar regional frameworks, which can set cross-border and data portability limitations. In addition, some countries’ national laws fall short in setting the required safeguards for data privacy breach and data portability and lack bodies such as data protection authorities to enforce these legislations.24 This is just the African scenario, but every country and region will have their own set of policies. Therefore, the healthcare programme and data controllers have an important responsibility to manage the programme in compliance with local laws or, where lacking, consult with government authorities on which regulation to abide by to foremost protect patient rights. The regulation of CDSS is also a very contentious topic, and relevant groups will have to watch the landscape closely to determine whether their tool fits the definition of a software as a medical device and what data they will need to generate for regulatory approval.
In terms of ‘Validation’, the big question was how to ensure CDSAs are safe for use. Developers and implementers should aim for clinical association, analytical validation and clinical validation. However, clinical validation can require significant amount of financial resources and time. As an example, e-POCT (electronic point-of-care test) was developed in 2014 to incorporate the latest scientific evidence to expand the medical content of IMCI. Its safety and efficacy were determined in an RCT in Tanzanian outpatient clinics.17 However, in 2019, the tool has not yet been implemented at scale or evaluated in non-controlled settings. Efforts like e-POCT to optimise CDSA content and assess its impact on patient health outcomes are needed to grow the evidence for CDSA’s role in improving patient care. However, providing these tools to patients also demands that we put our efforts in studies to measure the effectiveness of CDSA while in the hands of the end user and in the setting of intended use. Indeed, when CDSAs are implemented as part of routine practice, there may be factors that affect the overall effectiveness and impact of the tool. These include lack of or poor health worker training, lack of connectivity, software malfunctions, and poor design, leading to low usability and acceptability of the tool by health workers. Cycles of evaluation and improvement can be sought to perfect these tools in intended settings. In addition, digital data obtained from CDSA validation can be used in turn to dynamically improve the tool itself, faster than development improvements in classic wet diagnostics. CDSAs also have the potential to serve as platforms to support the evaluation or development of new content or guidelines. These can be adapted more quickly to different geographies to conform to local policies or support their update.
The call for the development of PHC during the Declaration of Astana revived the global commitment to provide quality health services to all.28 Achieving universal health coverage will therefore require that digital health interventions are quality-assured and evidence-based. Indeed, the WHO has begun developing frameworks to create a common digital language and synthesising the evidence around emerging mobile-based digital health technologies through two recent publications.8,29 These resources are meant to guide the global health community in assessing digital interventions that will improve quality of care, meaning interventions that are safe, effective, timely, efficient, equitable and people-centred, and these include CDSAs.30
Despite best efforts to balance expertise and knowledge as part of the multidisciplinary group to build consensus around a multicomponent tool, it is possible that some bias was introduced in certain sections due to a lack of expert input. A further limitation is that this TPP does not address some classic diagnostic characteristics, namely specimen type, performance accuracy (ie, sensitivity and specificity), operating conditions (ie, humidity and temperature) and cost. Since the tool is intended for an unspecified number of use cases due to the vastness of public health issues faced in healthcare systems, it was deemed expansive to define the multitude of different performance combinations. Further, TPPs usually include a cost estimate, but we could not estimate the cost of such a toolkit because again the use case and the setting of use are not fixed. The use of this kit would likely require a service delivery model attuned to the country of implementation.
Conclusion
The management of patients in low-resource settings with complex epidemiology is extremely challenging without clinical algorithms and accurate and portable rapid diagnostic tests. The consequences of suboptimal quality of care can go even beyond the patient’s health to include public health considerations, such as the development of antimicrobial resistance due to overprescribing of medicines. The aim of the proposed TPP is to support developers and implementers of these toolkits that are designed to guide health workers throughout clinical assessments, to give them confidence in clinical decisions and actions, whether it is sending a patient to a hospital or not, prescribing or not prescribing antibiotics, or recommending rehydration at home.
Acknowledgments
We would like to thank all the experts who participated in the FIND/WHO workshop to discuss the first draft of the TPP, as well as Rigveda Kadam, Wallace White and Mo Tobin (FIND) for discussions on TPP development. We would also like to thank the online reviewers who provided feedback on the development of the final consensus-driven TPP characteristics.
Footnotes
Handling editor Soumitra S Bhuyan
Contributors KGP and SD conceived the study. KGP generated the methodology, designed the survey and analysed the data. KGP produced the first draft of the paper. CR-A, VD’A, RS, FGM and SD provided edits and comments to the first draft. All authors reviewed and approved the final version of the manuscript.
Funding This work was funded by the Fondation Botnar and supported by the Global Antimicrobial Resistance Innovation Fund (GAMRIF), a UK aid programme. The funders had no role in the study design, data collection and analysis, or preparation of the manuscript.
Competing interests VD’A reports grants from Fondation Botnar, during the conduct of the study. In addition, VD’A has a patent free licence on a CDSA called ALMANACH.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement All data relevant to the study are included in the article or uploaded as supplementary information.