Evidence-to-decision frameworks: a review and analysis to inform decision-making for environmental health interventions

1. Scope

The focus of this paper is on the substantive criteria for decision-making with respect to interventions, including both normative criteria (what should be done) and feasibility criteria (what can be done).¹² Criteria related to the process (versus substance) of decision-making are beyond the scope of this work. While criteria in healthcare decision-making can be either qualitative or quantitative,¹⁰ this paper focuses on the former, although quantitative criteria (commonly used in HTA) were not excluded.

There are an increasing number of organizations which use systematic and transparent approaches to synthesize evidence in environmental health for hazard identification,¹³ risk assessment,¹⁴ and for the synthesis of the benefits and harms of interventions.¹⁵ These organizations were not included in the current analysis because they do not make recommendations on interventions to mitigate the effects of harmful exposures.

2. Bibliographic database search and article screening

To identify relevant EtD frameworks, we searched Medline via PubMed for systematic reviews of EtD frameworks, templates or tools published in English and indexed in the 10-year period up to 7 January 2021. The complete search strategy is found in Annex 1. We also solicited the advice of guideline development experts for any additional potential citations. Publications were included if they were systematic reviews of frameworks, tools or templates for formulating decisions or recommendations, or for priority setting related to the use of interventions (including diagnostic tests) in clinical medicine, public health, or environmental health. The included frameworks focused on population-level interventions; decision tools for the provider-patient interaction and patient decision aids were excluded. Frameworks which focused exclusively on economic considerations were also excluded. The setting for both decision-making and implementation of recommendations was not restricted.

Two persons (SLN and MTA) screened the titles and abstracts, and full-text versions were retrieved for studies which potentially fulfilled inclusion criteria. Consensus was achieved between the two reviewers for final inclusion in the review.

3. Search for EtD frameworks used by key organizations

in addition to the systematic review described above, we examined a broad range of organizations including clinical and public health guideline development groups in academia and the private sector, healthcare provider professional organizations, governmental agencies and international organizations. Given the purpose of this review, we focused particularly on organizations that work in the field of environmental health. Organizations were identified based on author knowledge of prominent guidelines in public health and clinical medicine, by snowballing (reviewing the origin of or basis for identified frameworks), and by conferring with a broad network of guideline developers internationally based on our experiences and contacts.

We used this pragmatic approach for several reasons. First, because of the dominance of the GRADE (Grading of Recommendations Assessment, Development and Evaluation) EtD framework over the last 15 years,^{5, 11, 16} many organizations either use GRADE or a modification thereof. Thus, a more exhaustive search was unlikely to yield novel or unique frameworks. Second, because the methods used by guideline development organizations are infrequently published in the peer-reviewed literature, an extensive hand-search of the grey literature including organizational web-sites would have been required, which was infeasible.

4. Data extraction and synthesis

For each systematic review identified in our search, a single author (SLN) extracted key information describing the focus of the review, time period searched, and the main findings, including an overview of the criteria identified across the included frameworks. We did not extract data on each individual framework included in each review.

For the EtD frameworks used by key organizations, we extracted data on organizational characteristics, the methods used to develop the framework, the funders and any declared interests of the developers of the framework, how evidence should be used to inform each criterion in the framework, quality assessment of individual studies and of the body of evidence, the specific EtD criteria, and the types of conclusions or recommendations formulated. One author extracted these data (SLN) and a second checked them (MTA); disagreements were discussed and consensus reached. Data were extracted into a template in Excel (Microsoft Corporation, Redmond WA, USA)

For the key organizations, the primary decision criteria for each EtD framework were extracted from the main published reports, and specifically from the identified template or list of criteria. For organizations for which there are significant background materials or publications such as for the GRADE framework, the decision criteria that were extracted into the tables and figure include those that are explicitly presented as part of the framework. Other concepts or criteria which are mentioned only in the narrative text accompanying a framework were not extracted unless the text suggested that they were important and were consistently applied in decision-making. Where a clear set of criteria could not be identified (e.g., criteria were presented in narrative text without clear demarcation of the main decision criteria, or when criteria appeared to vary across publications), the authors of the framework were contacted for clarification.

Data were summarized in a narrative, qualitative manner. Given that the organizations and frameworks discussed were a convenience sample, descriptive statistics and statistical comparisons are not meaningful. Quality assessment of the identified EtD frameworks was not performed as there is no standard for such an assessment.

1. Identified systematic reviews of EtD frameworks

The bibliographic database search for systematic reviews of EtD frameworks yielded 399 citations, of which four fulfilled inclusion criteria^{7, 10, 17, 18} (Table 1, Annex 2 and 3). One additional study which fulfilled inclusion criteria was identified by the co-authors.⁸ Each of these reviews included a cohort of EtD frameworks which the authors had systematically identified, with a particular focus: multicriteria decision analysis in HTA,¹⁰ vaccine adoption into national programs,¹⁷ diagnostic tests,⁷ decision-making in local low-income settings,¹⁸ and frameworks for informing health system coverage decisions.⁸ Each review summarized the main EtD criteria identified across their included, individual frameworks, and these main criteria are presented in Table 1. All five reviews included the balance of benefits and harms and consideration of resource use or cost-effectiveness. Only the review by Mustafa and colleagues⁷ included an assessment of certainty of evidence. Considerations of equity, acceptability and feasibility were included only in the reviews by Baltussen et al.¹⁰ and Burchett et al.¹⁷ The EtD criteria outlined in these five reviews generally corresponded to the main criteria in the EtD frameworks of the selected key organizations (see Results section 2). One additional publication, excluded from our review because it was not a systematic review, provides a useful list of “value assessment frameworks” for use in HTA for clinical interventions.¹⁹

2. EtD frameworks used by key organizations

Fourteen organizations that use EtD frameworks for recommendation formulation or decision-making were examined in detail (Table 2). One organization, the GRADE Working Group, is an informal network of individuals from a broad range of organizations, including academic institutions, national and international guideline development agencies, and healthcare provider organizations, among others.²⁰ The GRADE Working Group does not, itself, publish guidelines, but rather develops processes and methods for use by other organizations which develop guidelines. The other thirteen organizations develop guidelines for specific audiences and with a clearly defined scope or set of topics: five are agencies of national governments,^21–25 two are related to the World Health Organization (WHO),^{26, 27} one is a U.S. State agency (California Environmental Protection Agency (CalEPA)²⁸) and the remainder are non-governmental organizations or academic groups.^{2, 19, 29–31}

Two organizations focus primarily on clinical care: the Scottish Intercollegiate Guideline Network (SIGN)³² and the US Preventive Service Task Force (USPSTF).³³ The Guide to Community Preventive Services (GCPS) focuses on interventions aimed at groups, communities or health systems³⁴ and WHO and WHO-INTEGRATE primarily on public health interventions.^{26, 27} Three organizations are oriented to health technologies, with a prominent focus on economic evaluations and resource considerations.^{19, 30, 31} The UK National Institutes for Health and Care Excellence (NICE) examines a broad range of clinical, public health, and social interventions.³⁵ The Advisory Committee on Immunization Practices (ACIP) focuses exclusively on vaccine recommendations for U.S. populations.³⁶ Breast Cancer Prevention Partners (BCPP),²⁹ CalEPA,^{28, 37} and the Navigation Guide^{2, 15} focus on the human health effects of hazardous substances in the environment.

The amount of information available on each of the 14 key organizations varied greatly. There are a large number of publications describing the GRADE system for guideline development, including the GRADE EtD framework and its use.²⁰ On the other hand, there is only one document describing the approach of the BCPP.²⁹

Decision criteria

Eighteen frameworks were examined in detail: one from each of the 14 organizations, except for CalEPA and GRADE, where two^28,37 and four ^{6, 9, 11, 16} unique frameworks were examined, respectively. There were significant commonalities across frameworks in the criteria for formulating recommendations (Table 3 and Figure 1). Unsurprisingly, all included consideration of benefits, except the CalEPA framework for pesticides²⁸ which examines risks (of environmental exposures) and not benefits. All frameworks included an assessment of harms of the intervention under consideration. Fifteen of the 18 frameworks included some assessment of certainty or quality of the body of evidence across outcomes in the decision-making process (in contrast to an assessment of individual outcomes as part of an evidence review discussed below): BCPP, one of the CalEPA frameworks, and ISPOR (International Society for Pharmacoeconomics and Outcomes Research) did not.^{19, 28, 29} Some measure of costs, resource use or cost-effectiveness was found in all frameworks except BCPP, USPSTF and the GCPS.^{29, 33, 34} For the latter two frameworks, the U.S. government prohibits consideration of financial resources in the decision-making process. Other decision criteria were variably included: feasibility (13 frameworks) equity (12), values (11), and acceptability (11). Only two frameworks included human rights: WHO²⁶ and WHO-INTEGRATE frameworks.³⁸

• Download figure
• Open in new tab

Figure 1. Evidence-to-decision criteria for each key organization

Legend. Across the top of the figure, broad categories of evidence-to-decision criteria are presented. For each listed organization, the cell is shaded if their evidence-to-decision framework encompasses one or more criteria within a category.

Abbreviations:

AA, Alternatives Analysis

ACIP, Advisory Committee on Immunization Practices

BCPP, Breast Cancer Prevention Partners

CalEPA, California Environmental Protection Agency

EVIDEM, Evidence and Values Impact on DEcision Making

GRADE, Grading of Recommendations, Assessment, Development and Evaluation

GCPS, Guide Community Preventive Services

ICER, Institute for Clinical and Economic Review

ISPOR, International Society for Pharmacoeconomics and Outcomes Research

NICE, National Institutes for Health and Care Excellence

WHO, World Health Organization

View this table:

• View inline
• View popup

Table 3. Evidence-to-decision criteria for the key organizations

Criteria included in evidence-to-decision frameworks

Abbreviations: NI, not included

Footnotes

(^a) The criteria listed here are from both the “Considered judgement pro-forma 2014” form and the headings in the SIGN 50 guideline handbook as these differ.

“Priority of the problem” was included in the frameworks based on GRADE,^{26, 32, 36} as well as the BCPP,²⁹ EVIDEM (Evidence and Values Impact on DEcision Making),³⁰ ISPOR¹⁹ and Navigation Guide frameworks.¹⁵ However, in the background materials for several other frameworks, the burden of the disease (number of affected persons, severity of the disease, or economic and social costs) was mentioned as an important consideration when prioritizing an intervention for development of a guideline or policy. This latter consideration, however, differs from use of this criterion when deciding on the strength and direction of a recommendation.

The meaning of the criterion “values and preferences” has varied over time and across publications. This phrase encompasses two rather different constructs: the relative value that persons affected by the recommendations place on the outcomes of the intervention (values), and the preferences such persons have regarding the intervention options. The GRADE Working Group initially combined these two constructs⁵ with a definition encompassing both.³⁹ The 2016 update of the GRADE EtD framework distinguished these constructs, with the inclusion of two separate criteria: variability of the value affected persons place on the main outcomes, and acceptability of the intervention¹¹ to various stakeholders. The WHO EtD framework, published in 2014,²⁶ preceded the GRADE modifications of 2016,¹¹ includes “values and preferences” as well as “acceptability” and “feasibility”. The text explains that “values and preferences” “pertain to the relative importance people assign to the outcomes associated with the intervention or exposure; they have nothing to do with what people think about the intervention itself.”²⁶ Other key organizations include a decision criterion related to values using verbiage closely aligned with GRADE¹⁵ and focused exclusively on the relative value of outcomes,^{32, 35} or they include the concept but with a somewhat different meaning.^{19, 28, 36}

General aspects of evidence-to-decision frameworks

The process and methods for developing EtD frameworks were often superficial and poorly reported. Only WHO-INTEGRATE³⁸ and the GRADE coverage⁸ frameworks were developed following a systematic review of existing frameworks or potential decision criteria. Only WHO-INTEGRATE describes a conceptual framework²⁷ underpinning the EtD criteria. The lack of a conceptual framework represents a significant weakness in most EtD frameworks: key considerations may be missed and long-standing criteria may be perpetuated without adequate scrutiny.

GRADE has dominated guideline methods in health care and public health over the last 15 years. This has led to productive collaborations, standardization of processes and methods, and the development of packages of detailed methods and guidance by diverse contributors from across the globe. On the other hand, there are instances where intellectual dominance by a single group – i.e., monopolies of knowledge – can suppress innovation and slow down development processes.⁴⁶

EtD frameworks largely focus on, and are optimal for, single-component interventions with simple linear pathways from intervention to outcomes. In addition, most EtD frameworks do not consider the context in which the intervention is delivered. With the exception of WHO-INTEGRATE, EtD frameworks do not explicitly or even implicitly incorporate a complexity perspective including the inter-relationship between the intervention and the context or environment in which it is delivered. Only the GCPS, the USPSTF, and WHO-INTEGRATE recommend the use of visual depictions of the relationships among the intervention components, the causal pathway, and the context for the intervention (i.e., analytic frameworks, logic models, or similar depictions). Such approaches facilitate examination of multiple key questions linked across the causal pathway for intervention effects, as well as the consideration of contextual factors and externalities which may be important for decision-making.

Guidance on if and when to identify, synthesize and integrate evidence on criteria other than benefits and harms of the intervention into the decision-making process was provided only for WHO-INTEGRATE.²⁷ Most guideline development processes focus almost exclusively on benefits and harms in the evidence review and in discussions, without explicit efforts to prioritize other criteria or seek evidence to inform them. Qualitative evidence, particularly on considerations such as acceptability or feasibility, is rarely gathered. Meetings at which recommendations are formulated often include only brief, ad hoc discussions of these other criteria: this is a major limitation of most current approaches to recommendation formulation.

Evidence-to-decision criteria across frameworks

It was difficult to compare specific EtD criteria across frameworks due to significant variation in approaches to lumping and splitting criteria across frameworks, as well as in the terminology and definitions used. While most frameworks presented broad categories of criteria (e.g., “equity”), only WHO-INTEGRATE²⁷ provides detailed sub-criteria to facilitate understanding and decision-making. Some main criteria included several constructs: aggregating diverse elements within a single criterion can be difficult to implement in the decision-making context and when reporting the rationale for the recommendation or decision. For example, WHO-INTEGRATE’s “Human rights and sociocultural acceptability” includes both human rights and what other frameworks refer to as “preferences” or “acceptability”. CalEPA²⁸ articulates in one question a group of criteria: “Evaluating those options according to a value system that includes scientific, social, legal and economic factors, as well as practicality and enforceability”, without providing additional guidance or sub-criteria for decision-makers.

There is also variability in how criteria are defined and whether operational definitions or guidance are provided. The criterion “values and preferences” is particularly problematic. Unfortunately, guideline developers continue to use the phrase “values and preferences”, often with unclear and variable meaning. The WHO EtD framework, published in 2014,²⁶ includes an outdated definition of “values and preferences” which differs both from the 2008 GRADE guidance⁵ and from the 2016 modifications:^{11, 16} it needs to be updated.

The GRADE criterion of “priority of the problem” is also problematic. It was initially included in the GRADE framework to facilitate prioritization across interventions, such as at the national or sub-national levels. However, in most guideline development scenarios, the problem on which interventions and comparators are focused has already been determined to be of high priority. Thus, examination of the burden of disease and other priority considerations is irrelevant at the stage of recommendation formulation.

The criteria related to resource use vary considerably, and may include cost, affordability, infrastructure needs, personnel training, and measures of economic efficiency (e.g., cost-benefit and cost-effectiveness). This variability is due in part to the different perspectives of the sponsors of the EtD framework or of the target audiences for the organization’s products. For example, the USPSTF is prohibited by national legislation from considering cost or cost-effectiveness in their recommendations. On the other hand, for the UK’s NICE, cost-effectiveness is a critical part of their decisions. There is also variability in the extent and specificity of the guidance on how to incorporate resource use into decision-making. While well-developed for NICE,³⁵ and reasonably so for GRADE,⁴⁷ guidance is almost completely lacking for WHO²⁶ and WHO-INTEGRATE.²⁷

The criterion of “equity” is poorly described and little operational guidance is provided across the frameworks except for WHO-INTEGRATE which provides several sub-criteria for consideration under the main criterion of “health equity, equality and non-discrimination”.²⁷

Group decision making

Group decision making when formulating recommendations in guidelines is rarely a simple, linear process. While EtD frameworks support the normative aspect of decision making in terms of how expert panels ‘‘should’’ or ‘‘ought to’’ develop recommendations by providing a structure for decision making,^48–50 many factors affect how groups make decisions.⁴⁸ In addition to the features or characteristics of the decision itself (e.g., criteria in the EtD framework such as resource use and equity), other factors may play a role: i) situational or contextual factors (e.g., time pressure, social context, gender bias, political pressures); ii) individual characteristics of the decision maker (e.g., role on the expert panels, cultural and professional background, race/ethnicity, methodological expertise); and iii) individual panel member’s emotions and experiences (e.g., personal experiences with a disease). How these various factors contribute to decision making and recommendation formulation in guidelines is unknown,⁴⁸ but their potential effects must be kept in mind and made explicit to the extent possible.

While acknowledging the complex process of formulating recommendations in guidelines, EtD frameworks are a valuable tool and have led to vast improvements over “free-for-all” meetings where decision making criteria were selected in an ad hoc manner, and it was often unclear which considerations actually contributed to final decisions and their relative weight, what evidence was examined, and how and why the final recommendation or decision was arrived at.

Applying clinical and public health frameworks to environmental health

Applying approaches for clinical medicine interventions to other scientific fields, including environmental health, is challenging. There are important differences both in the assessment of the quality (certainty) of the body of evidence, and in translating evidence to recommendations or decisions. Woodruff and colleagues¹⁵ note that the GRADE system and other evidence-based medicine approaches have limitations in terms of applicability to questions and decision making in environmental health. The reasons include: i) the need to combine human, animal and (sometimes) mechanistic evidence; ii) the paucity of randomized controlled trials (RCTs) and other types of experimental studies in humans due to ethical considerations; and iii) differences in the decision-making context (e.g., weighing the benefits and harms is different for clinical interventions than for unintended exposures to substances in the environment).¹⁵ Nonetheless, the constructs of the GRADE and WHO-INTEGRATE frameworks likely apply to environmental health interventions.

Limitations of this analysis

The approach taken for this review and analysis has a number of limitations. The methods for identifying existing frameworks were limited: the systematic review included only reviews and did not include a search for all published EtD frameworks, and focused only on English language literature accessible through PubMed. Furthermore, many EtD frameworks are not published in the peer reviewed literature: rather, they are found on organizational web-sites and/or in organizational procedure manuals. For the review of key organizations, only a convenience sample was examined and EtD frameworks with important additional considerations or novel approaches may have been overlooked. However, it is unlikely that this approach missed key or significantly different frameworks in view of the dominance of GRADE and our consultation with experts on EtD frameworks.

Our analysis also has limitations related to the nature of the available data. Published information on the methods for developing frameworks was generally sparse, except for GRADE and WHO-INTEGRATE. The descriptions of the EtD criteria and how to operationalize them were also sparse for many frameworks. For some frameworks it was difficult to identify exactly which criteria were routinely included in the decision making process. While the publication might present a list or table of criteria, additional criteria might be mentioned in the adjacent text. However, the degree to which the guideline group or other decision maker addressed these additional criteria was often unclear.