Effectiveness of a multi-country implementation-focused network on quality of care: delivery of interventions and processes for improved maternal, newborn and child health outcomes

The Network for Improving Quality of Care for Maternal, Newborn and Child Health (QCN) aims to work through learning, action, leadership and accountability. We aimed to evaluate the effectiveness of QCN in these four areas at the global level and in four QCN countries: Bangladesh, Ethiopia, Malawi and Uganda. This mixed method evaluation comprised 2-4 iterative rounds of data collection between 2019-2022, involving stakeholder interviews, hospital observations, QCN members survey, and document review. Qualitative data was analysed using a coding framework developed from underlying theories on network effectiveness, behaviour change, and QCN proposed theory of change. Survey data capturing respondents perception of QCN was analysed with descriptive statistics. The QCN global level, led by the WHO secretariat, was effective in bringing together network countries governments and global actors via providing online and in-person platforms for communication and learning. In-country, various interventions were delivered in learning districts, however often separately by different partners in different locations, and disrupted by the pandemic. Governance structures for quality of care were set-up, some preceding QCN, and were found to be stronger and better (though often externally) resourced at national than local levels. Awareness of operational plans and network activities was lower at local than national levels but increased from 2019 to 2022. Capacity building efforts were implemented - yet often dependent on implementing partners and donors. QCN stakeholders agreed 15 core monitoring indicators though data collection was challenging, especially for indicators requiring new or parallel systems including those on experience of care. Accountability through community engagement, scorecards, and ombudsmen was encouraged but these initiatives remained nascent in 2022. Global and national leadership elements of QCN have been most effective to date, with action, learning and accountability more challenging, partner or donor dependent, remaining to be scaled-up, and pandemic-disrupted.


Introduction
156 Additionally, we adapted a psychometrically validated tool (5 domains, 40 indicators) developed for 157 evaluating clinical networks [18] to evaluate the network at national and local levels in each case study.
158 Over several rounds, we surveyed a variety of network members (e.g. clinicians, managers, advisors) 159 that also included QCN actors beyond our observation sites, totalling 1525 responses across the 160 countries and rounds of data collection (see [S2_Text] for a breakdown per case study). Respondents 161 had an option to fill in the survey online, via the Opinio platform, or on paper. Finally, we triangulated 162 the data collected with a document review that included all relevant published and unpublished 163 documents and communications relating to QCN at the global, national and district levels in the case 164 study countries. These included strategy and management documents, operational plans, directives, 165 formal minutes, and reports (see [S2_Text]). We were able to access unpublished documents via WHO 166 and Ministry of Health QCN contacts.

167
168 All qualitative data was analysed using a common coding framework developed from several 169 underlying theories that framed the overall QCN evaluation (see [S2_Text]). In this paper, our analysis 170 was guided by the QCN Theory of Change and monitoring framework[8] -the Leadership, Action, 171 Learning and Accountability (LALA) strategic objectives of QCN -and the environment, structure, 172 process and outcomes of the QCN. [19] All data was coded in NVivo 12, drawing on initial theory in 173 both an inductive and deductive way.
[17] Our codebook contained 'theory' codes related to 174 underlying theories; each theory was outlined using codes and sub-codes that broke down the 175 different components of the theory. The codebook was further supplemented by 'case study' codes 176 to distinguish data specifically relevant to each case study. We describe in more detail how the 177 codebook was developed, piloted and tested by researchers from the QCN Evaluation Group in 178 [S2_Text]. Over two years, many of our research team, including nine co-authors for this paper, were 179 involved in coding data in the different case studies. Six of the co-authors were also actively involved 180 in local data collection and familiar with the local context. Regular team meetings and rechecks took 181 place during each round of coding in order to ensure inter-coder consistency and that coders remained . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

Results
196 The QCN is a highly complex network in its composition, functioning and activities, operating at all 197 levels of governance. In other papers from this collection, we analysed the complexity of the network's 198 composition and functioning at global, national, sub-national and local levels. [7,10,[19][20][21] In this 199 paper, our analysis focuses on the effectiveness of the QCN through its various outputs and policy 200 consequences as well as through the impact of such activities. We begin our analysis at the global level 201 and then move to a cross-country analysis of QCN effectiveness at national and local levels in relation 202 to the four LALA strategic aims -Leadership, Action, Learning, Accountability -envisioned by the QCN 203 network as a whole. Table 1 summarises some of the key findings from our cross-country analysis.

204
. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted March 5, 2023. 222 where all global partners and network countries sent delegations of eight to ten people, creating 223 opportunities to evaluate progress of the network, share lessons learned and network between 224 countries. These meetings were seen as facilitating both the Learning and Accountability facets of the 225 LALA framework. Global and national participants noted that those meetings evolved in time as the 226 network matured, from the global leadership and technical experts sharing information 227 unidirectionally to national stakeholders, to countries sharing their progress and learning with one 228 another whilst global actors took a background role. Participants also saw the international meetings 229 as providing a necessary chance to "take stock", evaluate, and challenge the progress of the network.
230 Initially meant to occur annually, in practice only two international meetings have taken place: in . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted March 5, 2023.

299
. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review)

325
. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted March 5, 2023. ; https://doi.org/10.1101/2023.03.03.23286747 doi: medRxiv preprint 326 Leadership at national level has been important to coordinate and monitor QI efforts in the countries.
327 Funding for those functions however has been dependent on international donors, which impacted 328 the QCN implementation in-country and raises issues for the sustainability of leadership efforts. Even 329 in countries where network emergence was stronger [7] due to existing policies and initiatives and 330 where leadership and QoC were more institutionalised, effectiveness of the network was still 331 dependent on financial resources that are mostly external and on the commitment of governments 332 to dedicate resources to QoC. In Ethiopia, the government has a budget allocated for quality 333 improvement and QCN was part of it. This budget was small, especially after funds were diverted due 334 to the COVID-19 pandemic and to rehabilitate conflict-affected areas. Implementing partners 335 supported the 48 facilities by supporting coaching and learning activities and supporting the MoH to 336 prepare TWG and annual QCN meetings. In Bangladesh, efforts have been made to secure their own 337 independent budget but that has not been achieved yet according to government participants. 351 Others thought more detail was required to improve quality of care, for example, in Bangladesh: . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. 366 Learning sites were selected in each country (Table 1)  is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted March 5, 2023. ; https://doi.org/10.1101/2023.03.03.23286747 doi: medRxiv preprint 378 carrying out QCN activities in other districts they operate in. They further noted that this initial siloed 379 and disjointed implementation attempt slowed down the success of the network in Uganda and that 380 it was only three years into the five-year effort that a true network was beginning to form in the 381 country. Although frontline health workers in Uganda were more aware of quality improvement 382 efforts in general, led by the MoH and implementing partners, rather than QCN specifically. In

402
. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted March 5, 2023.  . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted March 5, 2023. ; https://doi.org/10.1101/2023.03.03.23286747 doi: medRxiv preprint 420 months was reported to have been spent directly on network activities, in Malawi 5-10 hours, Ethiopia 421 5-10 hours, and in Uganda 5-10 hours rising to 20-30 hours per 6 months (Figure 2). A greater 422 proportion of respondents in Bangladesh and Uganda also reported that their views and ideas 423 contributed to the network and that they were able to drive the network agenda (Figure 2).

461
"When you bring a five-year learning project, it should have its own resources for 462 implementation and to scale it up with the lessons learnt. This is the reason why the . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted March 5, 2023.

486
487 In Uganda, there was some attempt to pool resources from partners, but this effort failed because of 488 lack of trust, and a lack of harmonisation of different tools and methods used by different partners -. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted March 5, 2023. ; https://doi.org/10.1101/2023.03.03.23286747 doi: medRxiv preprint 489 both exacerbated by the leadership vacuum at the start of QCN in Uganda which later improved.
490 Additionally, as observed in several facilities, this has meant that when partners' projects came to an 491 end in a given area due to financial constraints or end of funding, partners ceased to support the 492 learning sites -making it difficult for sites to continue QCN activities given their lack of human, 493 technical and material resources, as well as undermining M&E efforts.

506
"Looking at the 8 standards for MNCH that we adopted, we realize that implementing 507 those standards isn't just about quality improvement. We need real inputs and we have 508 seen that without improving infrastructure, without having some of the services or the 509 equipment, for example the newborn care to improve it, we need to have actual 510 equipment, actual services, people need to have skills for managing those newborns. Some 511 of the inputs were not well catered for even in the 6 learning districts and probably that's 512 why we did not see any reduction in the newborn mortality. So, we have seen that the 513 input level as we implement it, the inputs that we plug into probably were not well 514 addressed. Even just the technical skills not just the quality improvement skills but the . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.  (Table 1), though experience of care indicators have been a challenge, 534 with independent regular systems of data collection from patients proving difficult. Data in learning 535 sites were often collected by implementing partners. In Bangladesh, both the government and the 536 major implementing partners aimed to place quality improvement indicators on the web-based 537 national dashboard, which can be used to download custom reports on various indicators. National 538 interviewees indicated that this dashboard was largely functional, and stakeholders expressed pride 539 in their ability to roll out this new initiative while simultaneously responding to COVID-19. In Ethiopia 540 on the other hand, the network relied on a parallel system to their HIS to collect data on the QCN . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted March 5, 2023. ; https://doi.org/10.1101/2023.03.03.23286747 doi: medRxiv preprint 541 quality indicators (data from facilities reported up in the system electronically via spreadsheets 542 provided by MoH) until a planned integration to DHIS2 at the next HIS revision. This proved to be a 543 challenge according to our interviewees who reported that the QCN indicators did not align with the 544 previous reporting system and questioned the robustness of the experience of care indicators since 545 the data was not collected by independent actors not involved in the care provided. However, all 546 countries faced structural issues and a lack of capacity in collecting good quality data. For instance, 547 countries still relied on separate surveys to collect data on experience of care indicators, unlike 548 indicators on provision of care that were captured by DHIS2 or a parallel system (for Ethiopia).
549 Additionally, in all countries, capacity building on data collection, analysis, quality and management 550 was crucial due to the lack of capacity at sub-national and local levels but depended on the efforts of 551 the partners supporting the learning facility. Supported by our facility observations, several 552 participants in all countries further brought up concerns over the reliability of the data reported, 553 particularly around patient experiences and mortality figures. The work of the QCN on indicators and 554 monitoring did bring attention to the importance of health data for QoC improvements. In Ethiopia 555 for example, national and local interviewees considered that the network had improved health data 556 documentation, management, use and reporting in health facilities, even if gaps in capacity remain. 557 In our survey, most respondents in Bangladesh (83% rising to 88%) and Uganda (74% rising to 80%) 558 indicated that there were quality improvement indicator dashboards or visualisations at their facility 559 ( Figure 1). This was 63% in Ethiopia and just over 50% in Malawi. Locally, in our observations in health 560 facilities, behaviour change around data monitoring seemed to vary. Some of the sites had a system 561 of bulletin boards which visually displayed QI metrics for the preceding several months, including 562 maternal and neonatal mortality, as a means of making the information more accessible and of 563 motivating staff. Some facilities, for example those observed in Bangladesh, also regularly completed 564 partographs to monitor the progress of labour as part of efforts to reduce mortality. In some facilities, 565 data was also discussed in the facility QI committee to improve accuracy and enhance accountability. 566 A few facilities were observed to not be using dashboards, visualising data or using monitoring . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity.

(which was not certified by peer review)
The copyright holder for this preprint this version posted March 5, 2023. ; https://doi.org/10.1101/2023.03.03.23286747 doi: medRxiv preprint 567 indicators in their practice. At the end of our data collection, there was still a lot of work to achieve in 568 all countries to collect (reliable) data, integrate to DHIS2, increase the quality of the data collected 569 and improve analysis at the national level. However, from our interviews and observations, the lack 570 of human resources and capacity around M&E makes this difficult to achieve. 578 579 As a result, the data received from countries was often incomplete or of poor quality leading to a lack 580 of quantitative analysis on how well the network has done on key process and outcome indicators and 581 whether the overall goal of the network of reducing case fatalities by 50% has been achieved.

571
582 Therefore, the impact of the network on reducing mortality remains unknown, though may be 583 reported in 2023 (WHO-Global Interviews-Round 2). For global interviewees, this goal was always 584 'ambitious' or 'aspirational' and they were cautious about the network's ability to achieve it within 585 the five-year timeframe. Among global actors there was an overall sense that the network needed 586 these bold, ambitious targets in order to gain momentum and attract engagement, funding, and global 587 attention. Most global actors believed these goals were an essential catalyst at the start of the 588 network and continued to serve as important motivation. Some national and local participants in 589 Bangladesh, Ethiopia, Malawi and Uganda believed that some progress had been achieved towards 590 reducing maternal and/or neonatal deaths in some of the learning sites. Our survey results indicated 591 that most respondents in each country perceived the network to be valuable, though more so in 592 Uganda and Bangladesh than in Ethiopia or Malawi (Figure 3). Figure 3 also shows a breakdown of . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted March 5, 2023. ; https://doi.org/10.1101/2023.03.03.23286747 doi: medRxiv preprint 593 indicators within this domain, with the highest scores being for "would recommend joining the 594 network", and the network "helps me professionally", and lower scores for perceiving QCN to have 595 resulted in healthcare improvements. CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted March 5, 2023. ; https://doi.org/10.1101/2023.03.03.23286747 doi: medRxiv preprint . CC-BY 4.0 International license It is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted March 5, 2023. ; https://doi.org/10.1101/2023.03.03.23286747 doi: medRxiv preprint