Assessing Knowledge, Attitudes, and Practices towards Causal Directed Acyclic Graphs among Epidemiologists and Medical Researchers: a qualitative research project

Background: Estimating the strength of causal effects is an important component of epidemiologic research, and causal graphs provide a key tool for optimizing the validity of these effect estimates. Although a large literature exists on the mathematical theory underlying the use of causal graphs, including directed acyclic graphs, to assess and describe causal assumptions, and translate these assumptions into appropriate statistical analysis plans, less literature exists to aid applied researchers in understanding how best to develop and use causal graphs in their research projects. Objective We sought to understand this gap by surveying practicing epidemiologists and medical researchers on their knowledge, level of interest, attitudes, and practices towards the use of causal graphs in applied epidemiology and health research. Methods We conducted an anonymous survey of self-identified epidemiology and health researchers via Twitter and via the Society of Epidemiologic Research membership listserv. The survey was conducted using Qualtrics and asked a series of multiple choice and open-ended questions about causal graphs. Results In total, 439 responses were collected. Overall, 62% reported being comfortable with using causal graphs, and 60% reported using them ‘sometimes’, ‘often’, or ‘always’ in their research. About 70% of respondents had received formal training on causal graphs (typically causal directed acyclic graphs). Having received any training appeared to improve comprehension of the


Introduction
Causal graphs, such as causal directed acyclic graphs, are a useful tool for improving the clarity of the assumptions required for valid causal inference, interrogating those assumptions, and designing appropriate data collection and analytic plans in light of the required assumptions (Greenland, Pearl, and Robins 1999;Tennant et al. 2020). Although the mathematical foundations of causal graphs have been well-specified, they have not been widely adopted in epidemiology and medical research (Greenland, Pearl, and Robins 1999;Hernan and Robins 2020;Pearl 1995;Tennant et al. 2020). We believe the reason for this implementation lag is the lack of available tools for the application of causal graphs in the study design and analysis phases of applied research studies. In anecdotal conversations with epidemiologists and other medical researchers, barriers to utilizing these models included uncertainty about which variables within a causal model to include in a statistical model, how to model complex or composite variables, and when to consider multiple time points or temporal ordering. While there are general mathematical axioms that provide some guidance on these questions, there are few resources that address them on a practical level.
The aim of this project was to characterize the knowledge, attitudes, and practices of epidemiologists and other medical researchers in relation to the use of causal graphs in their applied research studies, focusing on causal directed acyclic graphs (DAGs). We conducted a survey of practicing epidemiologists and researchers in closely related fields (e.g., biostatistics, medicine, health services research). The goals of this survey were: (1) to determine knowledge of the basic principles of creating and interpreting causal graphs; (2) to determine attitudes towards, experiences with, and practices of the use of causal graphs for designing applied research studies in order to identify needs and guidance; and (3) to identify perceived barriers to the use of causal graphs.

Participants
Our target population was researchers who work on applied epidemiology topics, whether or not they identify as epidemiologists, and who conduct at least some research in English. This is a large target population, and we were not able to enumerate the potential participants. Instead, we used two strategies to identify survey participants. First, we disseminated the request for participation through social media, in particular via Twitter, capitalizing on the large community of epidemiologists and medical researchers who use Twitter for academic conversations under the hashtag #epitwitter. Second, we contacted the Society for Epidemiologic Research, the American Public Health Association, the American College of Epidemiology, and the EpiMonitor newsletter to request our survey be sent out to their members/readers.

Survey distribution
The original Twitter link was available for four weeks. One week before the survey closed, we posted a reminder that the survey would be closing soon to encourage anyone with an incomplete survey to submit. All surveys were submitted by the time the survey link closed.
The Society for Epidemiologic Research posted the survey link to Facebook and Twitter, and the survey accessible from those links was available for six weeks. The other three organizations either did not respond to requests to distribute the survey or were unable to approve dissemination requests in the proposed time frame of the study.

Data collection and analysis
The survey was distributed using the Qualtrics platform and began with a description of the survey and informed consent materials. Participants who consented were asked to complete a section with basic demographic information, including gender, race/ethnicity, country where they work, type of organization . CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
is the (which was not peer-reviewed) The copyright holder for this preprint .
(academic, governmental/NGO, industry), and highest level of education. For all questions, participants were able to select the option 'choose not to respond' or submit a written answer instead of selecting from the list of options. The survey next asked a series of questions designed to assess training, knowledge, comfort with, and preferences for causal graphs in applied epidemiology. The survey was exempted by the Boston University School of Public Health Institutional Review Board. The surveys were completely anonymous and no IP addresses were collected. The full survey text is provided in the Supplementary Online Materials.
There were two open-ended survey questions asking participants to expand on their discomfort using DAGs and their attitudes about the usefulness of DAGs in a variety of research stages. We then used a grounded approach to code those responses into groups of similar responses, and systematically applied those groupings across all responses for both qualitative questions, using Microsoft Excel (Strauss 2013;Glaser 1967). Qualitative survey results were collated and major themes and supporting ideas were identified by study personnel. All results were stratified by source of participants (direct social media versus society dissemination).

Results
Of the 400 respondents who accessed the survey from the original Twitter link, 54% self-identified as women, 44% self-identified as men, 1% self-identified as gender-fluid or non-binary, and the remainder declined to report. Approximately 70% of respondents self-identified as white, 13% as Asian (including East Asian and South Asian), 6% as Hispanic, and 4% as Black (including African American and African). Table 1 shows the demographic breakdown of the participants. Overall, 34% of respondents were current students, and women were somewhat more likely than men to be current students -38% of women versus 28% of men. Approximately 54% of respondents reported their highest completed degree was in epidemiology or that they were currently enrolled in an epidemiology program. Most participants worked in academia and about half of respondents stated the US was their primary place of work.
There were 39 respondents from the Society for Epidemiologic Research-disseminated survey. Approximately 58% self-identified as women and 42% self-identified as men. Seven participants declined to report. About 65% of the respondents identified as white, 15% as Asian (including East Asian & South Asian), 8% as Hispanic, and 3% as Black (including African American and African). About 39% of participants were students, and most studied or were currently studying epidemiology. Most respondents reported working in academia and 44% of respondents identified the US as their primary place of work. These demographics generally matched those of the population who responded to the original Twitter link.
Due to the limited number of responses to the Society for Epidemiologic Research-disseminated survey link, and because we could not verify that there was no overlap in respondents (since we did not collect IP addresses), the remainder of this paper will focus primarily on the results from the first, Twitter-based, version of the survey.

Knowledge
Most of the Twitter respondents had received formal training in causal graphs (69%). Most of these participants received training from graduate school courses (79%). Of these respondents, 93% said they received training on the use of directed acyclic graphs (Greenland, Pearl, and Robins 1999), 16% on the use of single world intervention graphs (Richardson and Robins 2013;Robins and Richardson 2010) and 7% on the use of finest fully randomized causally interpretable structured tree graphs (Robins 1986).
Responses to knowledge assessment questions were mixed (Table 2 and Table 4). About 51% of respondents correctly identified that not drawing an arrow between two variables was a stronger . CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
is the (which was not peer-reviewed) The copyright holder for this preprint . assumption than drawing an arrow (35% chose the wrong answer and 14% were unsure). Respondents who reported having received any training on causal graphs were more likely to choose the correct answer for this question (58%) and were less likely to be unsure (9%) than individuals who reported no training (35% correct; 27% unsure). Interestingly, while individuals who had completed an epidemiology program or who were current students in an epidemiology program were more likely to choose the correct answer than individuals in other fields, the type of training (graduate school course, workshop, or online training) did not appear to be associated with level of knowledge. When asked to choose which of two directed acyclic graphs (Figure 1), if true, allowed identification of the causal effect of post-traumatic stress disorder on suicide, 62% of respondents chose the correct graph (this question assessed knowledge about the depiction of measurement error in DAGs). There was essentially no difference in performance on this question between individuals who did and did not report having received training in causal graphs. However, current students were more likely than non-students to choose the correct answer (71% vs 58%) ( Table 2).

Attitudes
Most respondents somewhat or strongly agreed that they were comfortable using causal graphs for designing data collection (62%), identifying appropriate adjustments sets (66%), evaluating existing studies (59%), and assessing surprising study results (59%), regardless of training history. Nearly 80% of respondents somewhat or strongly agreed that causal graphs were useful in a classroom setting for describing bias, in an applied research project at the study design or analysis phase, or in reviewing a paper or critiquing an existing study. Over half (67%) of participants indicated that a more complicated DAG was best for designing a research study ( Figure 2c) and least useful for presenting study assumptions (26%). The simpler DAGs shown in Figure 2a and Figure 2b were rated as being more useful for presenting results (34% and 39% respectively).
Effect measure modification and interaction are often of interest in epidemiology, but there is no clear guidance on visualizing and assessing these features in a causal DAG (Hernan and Robins 2020). Several schools of thought exist on the appropriate approach, including that (1) arrows are agnostic to the presence or absence of effect modification and interaction and therefore all arrows potentially represent these relationships (Greenland, Pearl, and Robins 1999); (2) effect modification and interaction arise from the presence of alternate causes of the outcome of interest that vary in distribution by exposure status, and so can be displayed by including unknown common causes of exposure and outcome (VanderWeele and Robins 2007; Hernan and Robins 2020); and (3) effect modification can be shown by drawing an arrow from the modifier to another arrow, instead of a node (note that these are no longer DAGs, and should be referred to as causal graphs) (Weinberg 2007). When we asked respondents "to the best of your understanding, can effect modification be represented in a directed acyclic graph," there was substantial disagreement (Table 3). Interestingly, those with no training were much more likely to choose 'don't know' than those with any training (49% vs 11%), and less likely to choose 'no' or 'yes, with modification of the traditional rules' (17% and 19% for those with no training; 32% and 37% for those with any training, respectively) ( Table 3).

Practices and barriers
About 60% of participants responded that they used causal graphs in an applied epidemiology project (defined as reporting use at least sometimes). Of these participants, only 20% responded that they used systematic processes (literature reviews or expert consensus) to develop a graphical causal model. The remaining 80% either performed informal literature reviews and conversations or re-used already published models. The most common reason for not using a graphical causal model in applied epidemiology research was lack of knowledge on how to use them (40% of responses). When it came to challenges specifically associated with building causal graphs in epidemiology research, the most common challenge was choosing which arrows to omit (56% said this was the most challenging or second most challenging task), followed by choosing which arrows to include (45% as first or second most . CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv a license to display the preprint in perpetuity. is the (which was not peer-reviewed) The copyright holder for this preprint . challenging). In terms of assessing causal graphs in research, participants found identifying potential unknown sources of error and identifying collider bias to be the two most challenging aspects (64% and 55%, respectively).
Respondents who regularly use causal graphs ("sometimes", "often", or "always" responses) in their research most frequently said reference material, software tools, and availability of pre-published causal graphs would be of most use in helping them decide to use causal graphs more often (16%, 15%, and 15% respectively). On the other hand, respondents who did not use causal graphs ("rarely", "once", or "never" responses) said online training would be most valuable, followed by in-person training and software tools (22%, 17%, and 16%). Approximately twice as many respondents who reported not regularly using causal graphs indicated that journal or grant agency requirements to use causal graphs would be useful for their uptake of these methods, compared to respondents who reported regularly using causal graphs (10% versus 5%).
Individuals who reported that they did not feel comfortable using causal graphs for designing data collection, identifying adjustment sets, evaluating studies or assessing surprising study results were given the opportunity to explain what aspects they were not comfortable with and what resources might help. A common theme among the response was lack of knowledge contributing to discomfort using causal graphs. One participant wrote, "I haven't received training in them at all but I feel like I should know more about them." Other responses included more specific knowledge gaps. For example, one participant expressed concern about "How to incorporate effect modification, simultaneous bias, measurement error," and another said, "Knowledge about how to correctly use the arrows." Additionally, many participants wanted resources and training including a "resource library," "access to experts," and an "online training materials." Participants were also asked to expand on the usefulness of causal graphs in classrooms, designing research, analyzing results, and reviewing completed work. Many respondents addressed a lack of knowledge about causal graphs, with some general statements such as "I should study it more" and "I do not understand how it would be used in this context." Some participants mentioned more specific gaps in knowledge including "how to depict moderation." Additionally, several respondents expressed concern about the methodological components of causal graphs including that "some forms of bias are not amenable to being described by DAGs" and "molecular biology is often bi-directional which is never represented." There were also a few participants who expressed concern over using DAGs for analysis, stating "analyses have to be done based on model formulas, not on DAGs" and "at the study analysis phase you stick to the analysis plan."

Society for Epidemiologic Research survey results
In general, the responses from the Society for Epidemiologic Research-disseminated survey link were similar to those from the original Twitter survey. The racial, ethnic and gender self-reported information was similarly distributed in both survey versions. The majority of respondents in both surveys were not students and most worked in or held degrees in epidemiology. Most of the respondents in the Society for Epidemiologic Research-disseminated survey link reported using causal graphs in their research at least "sometimes" (68%). Of these participants, over 90% reported using informal methods (literature reviews and conversations) or re-using already published models to develop models for their research. Aligned with the original Twitter survey responses, reference materials were cited as the most useful tool to increase DAG usage among those who already use DAGs in research in the Society for Epidemiologic Research-disseminated survey. The most notable difference between the two surveys was that a higher proportion of respondents of the Society for Epidemiologic Research-disseminated survey link correctly answered that not drawing an arrow between two variables was a stronger assumption than drawing an . CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
is the (which was not peer-reviewed) The copyright holder for this preprint . https: //doi.org/10.1101//doi.org/10. /2020 arrow (73% compared to 50% in the original survey). Holding an epidemiology degree and being a student similarly did not appear to be associated with the correct response rates (Table 4).

Discussion
While a large proportion of participants who professionally engage in applied epidemiology reported DAGs were useful, a smaller proportion used them in their research or said they were comfortable using them. Additionally, there was disagreement about the representation of effect measure modification in DAGs, signifying that there may be confusion about this within the research community. The most common reason for not using DAGs in applied epidemiology projects was lack of knowledge. Those who did not use DAGs thought online or in-person trainings, as well as software tools, would help them use causal graphs more often.
Our findings build upon the findings of a recent, large, systematic review of published causal graphs in the epidemiologic literature (Tennant et al. 2020). There, the authors found that 38% of papers which claimed to have used DAGs did not include the model in the main text or supplementary material. Of the DAGs that were published, there were significant variations in terms of model design (i.e., number of nodes/arrows and use of unobserved variables) and DAG-based adjustment techniques. This may be explained, in part, by our finding that preference for the design of a research DAG varied depending on the purpose of the DAG (whether for designing a research study or for presenting study assumptions) ( Figure 2). Additionally, the authors reported that extraction errors were reduced when diagrams were constructed using DAGitty, a web-based platform for designing and exploring causal diagrams (Textor et al. 2016).

Future directions.
Our results demonstrate a need for more resources and guidance on the application of DAGs to epidemiology research. Although interest in causal graphs is high, the lack of available training resources possibly is a major barrier to use. Promisingly, we saw relatively little difference in the level of knowledge about DAGs among those who had received graduate school training, workshop training, or online training, suggesting that increasing the availability of even a relatively low-intensity training may have substantial impact on the knowledge, and thus use of, causal graphs in epidemiology research. Finally, our results also suggest that improved guidance on the visualization and assessment of effect measure modification or interaction in causal graphs is necessary to address the uncertainty about this topic. Future research should also be focused on identifying explicit topics (i.e., expressing complex mixtures in DAGs) and training resources for researchers interested in using DAGs for study design and analysis.

Limitations.
A potential limitation of this study is that the sample is unlikely to be representative of all epidemiologists and medical researchers. In particular, it appears that women may have been somewhat more likely to participate than men, and current students may be over-represented. The sample of participants who completed the survey may not be generalizable to the population of all practicing epidemiologists, especially since the Society for Epidemiologic Research was the only organization that responded to our request within the given time frame. The subset of epidemiologists who use Twitter may be younger than those who do not, and, if so, may be more likely to have been exposed to causal graphs during their graduate training. However, we are aware of no previous research on the application of causal graphs among epidemiologists, and as such this study will be a valuable guide to the development of more systematic assessments and the creation of tools to improve the implementation of causal graphs in applied epidemiologic research.

Conclusion
Overall, there is an interest in graphic causal models within the epidemiologic community; however, a lack of training resources presents a barrier for their widespread use in practice. Generally, epidemiologists and medical researchers who received training in causal graphs were shown to have better knowledge about the underlying assumptions for building DAGs compared to those without any training, regardless of the intensity of training. Easily accessible resources that provide guidelines for creating evidenced-based DAGs are likely needed to increase DAG utilization, which may improve analytic adjustment for confounders in published epidemiologic research. . CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
is the (which was not peer-reviewed) The copyright holder for this preprint . Table 2. Responses to the question "Which of the following directed acyclic graphs, if correct, would imply that the true causal effect of post-traumatic stress disorder (PTSD) on suicide could be estimated without error? (select all that apply)" by student status, degree field and training. The assessed causal DAGs are shown in Figure 1. For both surveys, the correct choice was DAG B (Figure 1b) is the

(which was not peer-reviewed)
The copyright holder for this preprint . . CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
is the (which was not peer-reviewed) The copyright holder for this preprint . https://doi.org/10.1101/2020.01.17.20017939 doi: medRxiv preprint Figure 1: Suggested directed acyclic graphs (DAGs) demonstrating assumptions about measurement error for depression and post-traumatic stress disorder (PTSD). Survey respondents were asked "Which of the following directed acyclic graphs, if correct, would imply that the true causal effect of post-traumatic stress disorder (PTSD) on suicide could be estimated without error? (select all that apply)". For response summary, see Table 2.
Figure 2: Three approaches to drawing directed acyclic graphs (DAGs) for study design, and presentation of research assumptions. Survey respondents were asked to rank the options in order from most (1) to least (3) useful for presenting research results and from most (1) to least (3) useful for designing the study. The goal of this question was to assess attitudes towards node complexity in causal DAGs. ? in . CC-BY 4.0 International license It is made available under a author/funder, who has granted medRxiv a license to display the preprint in perpetuity.
is the (which was not peer-reviewed) The copyright holder for this preprint . https://doi.org/10. 1101