Refining clinical algorithms for a neonatal digital platform for low-income countries: a modified Delphi technique

Study design

This study used a two-step modified Delphi technique.[13] The Delphi technique was chosen because it is an effective method of gathering expert knowledge from geographically diverse leaders in the field to address complex clinical problems that lack evidence.

Recruitment

Twenty-two neonatal experts were invited to participate in the study. This number represented an adequate sample size[14, 15] and permitted a manageable amount of data collection. Participants were recruited if they were a physician or neonatal nurse practitioner with more than ten years neonatal experience (at least three in LICs), neonatal postgraduate training, fluency in English, internet access and willingness to participate. Neonatal experts known to the researchers for their clinical expertise, research and contributions to guideline development in LICs were identified in equal numbers from both LICs and HICs. No financial incentive was offered, but reimbursement for costs of Skype calls was provided for some experts in LICs.

Algorithms and item generation

The four clinical decision algorithms selected for review were neonatal sepsis, hypoxic ischaemic encephalopathy (HIE), respiratory distress of the newborn and hypothermia. These conditions represent the leading preventable causes of neonatal mortality and are the most difficult to diagnose and manage appropriately in LICs with some of the weakest WHO GRADE recommendations and quality of evidence.[16] For example, the European definition of neonatal sepsis is two or more clinical symptoms and two or more laboratory signs in the presence of, or as a result of, suspected or proven infection.[17] This definition is not possible in LICs where laboratory investigations are not routinely available.

Items were identified by comparing the algorithms side-by-side with the international (WHO) and local neonatal guidelines (Care of the Infant and Newborn in Malawi - COIN) from which they had been derived. This comparison generated a comprehensive list of items where discrepancies in diagnostic parameters and treatment recommendations required expert opinion. Once finalised, the clinical algorithms and list of items (henceforth referred to as questionnaire) were piloted with two paediatricians with neonatal experience in LICs. Ambiguous items were amended accordingly.

Delphi technique

The questionnaire was circulated by email to the experts with specific instructions at least two weeks before they were interviewed. Each algorithm was verbally and diagrammatically explained with their references specified (i.e., WHO, COIN or NeoTree research team) to aid in decision-making during the interview (online supplementary file 1). Round one interviews were conducted in June and July 2018. Experts were sent up to two reminder emails to schedule their phone or Skype interview. Interviews were conducted privately from a home office. Standardised questions were used to review each item from the questionnaire. Experts were asked to rate their level of agreement for each item using a five-point Likert scale.[18] Each rating was followed by open-ended questions to obtain the experts’ rationale for their response and any amendment or additional items they would propose. All interview data were transcribed using both audio- recordings and notes made during the interview by the facilitator. All responses were anonymised (with participant numbers) and reviewed together with the quantitative results. Based on sample size, the level of agreement for consensus opinion from the panel was set at 80%.[14] Items that met consensus (≥80% agreement) were included or were modified with minor changes to wording based on expert advice. Items that did not meet consensus (<80%) were removed or modified according to the feedback from the expert panel and submitted for the second round. Items that did not meet consensus were still included if they were part of WHO and COIN guidelines so that frontline HCWs continued to follow the current standard of care. A second questionnaire was designed with modified items and expert additions from the first round.

In round two (June and July 2019), this second questionnaire was distributed electronically to the 14 experts from round one (online supplementary file 1). A results summary from round one was sent to the experts, and the full set of anonymised results were made available at their request. Two email reminders were sent to non-responders. Experts again rated items on a Likert scale and explained their ratings. Responses were analysed as described in round one, and items meeting ≥80% consensus were kept for the final NeoTree algorithms.

Patient and public involvement

While key stakeholders were involved in co-developing the NeoTree digital platform, there was no patient or public involvement in this Delphi study.

Consent procedures

The goals and processes of the project were explained to the experts in their email invitation, and consent was obtained by email agreement. Experts were verbally informed at the beginning of the first round that their responses would be kept anonymous.

Round One

Fourteen experts (63% response rate) completed round one. Interviews averaged 73 minutes (40- 110 minutes). Thirty-four items (45%) reached consensus (Figure 1). These items were either: (1) included, unmodified (32%); (2) included, modified (11%); or (3) changed for clarification in the second round (2%). Items that did not reach consensus (55%) were either: (1) excluded from the revised algorithm (30%); (2) included because they were part of WHO/COIN guidelines or the Thompson score (11%);[19] or (3) changed and submitted for a second round (14%) (Table 2).

• Download figure
• Open in new tab

Figure 1.

Outcome of algorithm items after round one and round two of the Delphi technique.

Abbreviations: COIN = Care of the Infant and Newborn in Malawi; MCTW = minor changes to wording; WHO = World Health Organisation

The expert panel consistently stated that algorithm items must comply with WHO danger signs and COIN guidelines for neonatal sepsis, irrespective of whether the panel agreed with them. For example, experts thought that ‘bulging fontanelle’ was “subjective; there are non-infectious causes … many babies’ fontanelles bulge when they just cry.” Another item that did not meet consensus was ‘poor feeding,’ which experts found vague for multiple reasons, including: “it is too subjective; it depends on how long for … many newborns do not feed well on the first day of life.” However, experts agreed that poor feeding was a sign of possible sepsis if it was “a new onset of poor feeding when the infant had previously been feeding well.” This item was changed to ‘new onset of poor feeding’ for the final algorithm.

Two items that were included because they are part of the COIN guidelines highlighted inconsistencies with WHO guidelines. For example, COIN uses a temperature of more than 37.5°C as a fever for a newborn, whilst WHO and most experts use more than 38°C. Therefore, 37.5°C was included for the Malawian digital platform, but 38°C will be used for other countries. Other items where a difference between the recommendations and the guidelines occurred were antibiotic choice and duration for neonatal sepsis.

Modifications usually involved adopting the language utilized by WHO or COIN, but there were items that experts felt needed clarifying. For example, experts felt that “twitching or abnormal movements” needed to be added to the WHO term ‘convulsions’ because seizures in a neonate can be very subtle. Certain items that that could not be revised easily were submitted for the second round according to feedback from the expert panel. For example, experts disagreed that ‘very/extremely premature (<32 weeks gestation)’ was a major risk factor for sepsis if “the baby was delivered as a clean cold caesarean section for maternal reasons and the mother was not in labour.” Eighty percent of experts highlighted that the algorithm should include weight to guide gestation because “gestation is often unknown” and “you are relying on [the] Ballard score which has plus or minus two weeks accuracy.” Similar opinions regarding method of delivery and the importance of birth weight were expressed for ‘slightly premature (32-36 weeks gestation).’ Both gestational age brackets were submitted into the second round as risk factors for sepsis after modifying the items to include WHO weight parameters to guide gestation. Other items that did not gain consensus and were submitted for the second round included items that experts felt needed further clarification. ‘Born before arrival’ as a minor risk factor for sepsis was clarified to the experts that this meant the baby was born en-route to the hospital (either in a vehicle or on the roadside, both being considered dirty environments in Malawi). A ‘neonate admitted with or history of a fever’ as a minor risk factor for sepsis was changed to ‘mother reports a non-measured fever’ in the second round. Lastly, because experts considered the term ‘birth injury’ unclear, we asked them in round two to define what they considered a ‘significant birth injury.’

Key findings by algorithm

The first important finding was that the ‘major’ or ‘minor’ algorithmic weighting system (where one major risk factor for sepsis is equivalent to two minors) used to diagnose neonatal sepsis was near consensus (77%) but did not meet the 80% threshold. Experts called for further evidence before adopting this system: “It is a difficult thing to do…you need to work out how specific and sensitive the app is by looking at blood cultures.” Two experts suggested using a risk score calculator, and another two experts highlighted that WHO only uses danger signs. This weighting system was subsequently removed from the algorithm (Figure 2).

• Download figure
• Open in new tab

Figure 2.

Modification of the neonatal sepsis algorithm as a result of the Delphi technique.

Abbreviations: AB = antibiotics; HAI = hospital-acquired infection; IVAB = intravenous antibiotics; LICs = low-income countries; > n = number of risk factors to be determined

The second significant algorithmic finding was on HIE. An academic expert in neonatal encephalopathy discouraged the use of the term ‘birth asphyxia,’ a term used by Malawian HCWs and therefore incorporated into the original algorithm.

You really must not call it birth asphyxia because birth asphyxia means failing to breathe at birth and what you are talking about is encephalopathy.

Additional feedback on the algorithm focused on the combination of risk factors or clinical signs and symptoms to consider or diagnose HIE. Experts cited a lack of evidence for using risk factors to diagnose birth asphyxia and that the application should only be using clinical signs and symptoms.

Birth asphyxia is not about risk factors. If you have encephalopathy it is a clinical diagnosis, and it is irrelevant what your risk factors are.

Experts recommended utilizing a validated encephalopathy score,[19, 20] which was incorporated into the HIE algorithm. The risk factors that met consensus will be used as prompts to perform the Thompson score, which uses clinical signs and symptoms exclusively to diagnose HIE.

Third, for the respiratory algorithm, experts highlighted that “it is hard to make an accurate diagnosis of a respiratory condition without investigations.” Therefore, the algorithm should focus instead on the management of respiratory distress. All respiratory conditions (respiratory distress syndrome, meconium aspiration, congenital pneumonia and transient tachypnoea of the newborn (TTN)) now fall under the umbrella diagnosis of respiratory distress of the newborn within the algorithm. For teaching purposes, the four respiratory conditions will be included as ‘diagnoses to consider’ in the management.

Finally, for the hypothermia algorithm, experts commented that first-line treatment for all newborns be skin-to-skin care including those who were severely hypothermic (<32°C) unless they showed any signs or symptoms of being unstable. Additionally, experts did not think it was realistic to review a newborn every 15-30 minutes when hypothermic.

Round Two

Ten (71%) experts completed round two, seven electronically and three by telephone interview. Four experts dropped out (three from HICs, one from LIC); three did not respond to email reminders and one expert was unable to meet the completion deadline. Nine items (41%) reached consensus (Figure 1). These items were either (1) included, unmodified (36%) or (2) included, modified in the revised algorithm (5%). Items that did not reach consensus (59%) were either (1) excluded (41%) or (2) included, modified according to WHO guidelines or expert suggestion (18%).

In round one experts indicated that hypothermia was a major sign of sepsis and should be included in the sepsis algorithm if persistent. In round two we clarified that the application is to be used at the time of admission onto the neonatal unit, at which point the HCW will only have one temperature reading. Experts in round two disagreed that a single temperature reading of <35.5_JC was a sign of sepsis and felt that it would more likely be due to environmental hypothermia, a common problem in LICs. Additionally, in round two it was established that experts were much more concerned with extremely premature <1500g neonates (88% consensus) compared to slightly premature 1500-2500g neonates (62% consensus) being at risk for neonatal sepsis. Central cyanosis was an addition to the second round as an expert suggestion to include all WHO danger signs; despite missing consensus (with 78% agreement), it was ultimately included in the final sepsis algorithm to comply with WHO guidelines for danger signs. All of the respiratory items marginally missed consensus. All three items were still included in the final teaching algorithms with revisions in line with WHO diagnostic criteria. Modifications to the algorithms can be found in online supplementary file 2.