Abstract
The role of genomics in healthcare is expanding rapidly and many countries are set to explore the possibility of using genomic sequencing to expand current newborn screening programmes. Offering routine genomic newborn screening (gNBS) would allow newborn screening to include a much broader range of rare conditions, but there are many technical, practical, psychosocial, ethical and economic challenges to be addressed. Genomics England and NHS England have established the Generation Study to deliver gNBS for 100,000 births in 2024/5 to explore the benefits, challenges, and practicalities of offering gNBS to parents in England. Here we describe the study protocol for the Generation Study - Process and Impact Evaluation, an independent mixed-methods evaluation of the Generation Study. The evaluation will have oversight from a Study Advisory Group that includes academic, clinical and patient representatives and a Patient and Public Involvement and Engagement (PPIE) Advisory Group that includes members from parent and patient organisations and parents with relevant experiences. The Process and Impact Evaluation will examine whether offering gNBS in routine care is feasible and acceptable and inform our understanding of the clinical utility and cost effectiveness of gNBS in England. Through surveys and interviews we will explore the attitudes and experiences of parents, professionals and patient organisations. We will also consider the clinical, psychosocial and health economic impacts, both positive and negative. The results will be presented at national and international conferences and submitted for peer review and publication.
SUMMARY
The Generation Study - Process and Impact Evaluation aims to provide an independent mixed-methods evaluation of the use of genomic newborn sequencing (gNBS) for the early diagnosis of rare, childhood-onset, actionable genetic conditions.
The evaluation comprises seven core studies:
Study 1: Identifying goals, challenges and early lessons in implementation
Interviews with professionals who are the Generation Study’s “designers” and “early implementors”; shadowing of processes and pathways; and documentary analysis.
Study 2: Impact, experiences and attitudes of parents
Survey and follow-up interviews with parents following receipt of gNBS results to assess acceptability, experience, attitudes and impact (positive and negative).
Study 3: Gathering wider professional viewpoints about the Generation Study and gNBS implementation
Survey with staff delivering the Generation Study at early adopter sites and professionals from a range of relevant backgrounds across England.
Study 4: Views of the rare disease community
Surveys and interviews with advocates from rare disease support groups to examine attitudes to gNBS including positive and negative impacts as well as unintended consequences on parents and the rare disease community more broadly.
Study 5: Public views
Survey conducted with a market research company to capture views of with a diverse group of parents in terms of age, gender, ethnicity, socio-economic status and geographical location to evaluate the key findings around acceptability and attitudes towards gNBS with parents who have not been involved in the Generation Study.
Study 6: Cost effectiveness evaluation
Assessment of the impact of gNBS on healthcare resource use and costs, quality of life outcomes and non-health-related outcomes, to inform the Genomics England health economics model.
Study 7: Clinical utility assessment
Assessment of the clinical utility of gNBS for health-related outcomes by comparing hospital contact and mortality rates in various groups of children (confirmed diagnosis, population controls), making use of the longitudinal healthcare data linked to the Generation Study and control groups primarily drawn from ECHILD.
Summary Table: Research Questions, Methods and Outputs
AIM
To conduct a process and impact evaluation on the use of genomic newborn screening for the early diagnosis of rare conditions that can be treated in early childhood.
PRIMARY RESEARCH QUESTIONS
1. Is the use of whole genome sequencing (WGS) as a tool for early diagnosis of rare, childhood-onset, actionable genetic conditions feasible and acceptable?
2. What is the impact (both positive and negative) of the programme on stakeholders and the wider system?
3. What are stakeholders’ experiences and attitudes to the use of WGS as a tool for early diagnosis of rare, childhood-onset, actionable genetic conditions?
4. What are the implementation processes being used to deliver the Generation Study and how do these vary across sites?
RESEAERCH TEAM
Dr Celine Lewis, UCL Great Ormond Street Institute of Child Health
Professor Cecilia Vindrola-Padros, UCL Rapid Research Evaluation and Appraisal Lab Sigrún Clark, UCL Rapid Research Evaluation and Appraisal Lab
Katie Gilchrist, UCL Rapid Research Evaluation and Appraisal Lab Professor Felicity Boardman, University of Warwick
Dr James Buchanan, Queen Mary University of London Mr Martin Vu, Queen Mary University of London
Professor Pia Hardelid, UCL Great Ormond Street Institute of Child Health Dr Ania Zylbersztejn, UCL Great Ormond Street Institute of Child Health Dr Amy Hunter, Genetic Alliance UK
Dr Jennifer Jones, Genetic Alliance UK
Mrs Kerry Leeson-Beevers, Alström Syndrome UK and Breaking Down Barriers Dr Melissa Hill, Great Ormond Street Hospital NHS Foundation Trust
Ms Bethany Stafford-Smith, Great Ormond Street Hospital NHS Foundation Trust Mrs Wing Han Wu, Great Ormond Street Hospital NHS Foundation Trust
STUDY OVERSIGHT
The Process and Impact Evaluation will have oversight from two advisory groups:
Study Advisory Group
The Study Advisory Group, chaired by Dr Rachel Knowles (UCL Great Ormond Street Institute of Child Health), with members from academic, clinical and PPI backgrounds. PPI members will be from relevant patient organistations. The Study Advisory Group will oversee and guide the study. They will meet every quarter to review study progress and provide guidance.
Patient & Public Involvement (PPI) Advisory Group
A PPI Advisory Group, chaired by Kerry Leeson-Beevers (Breaking Down Barriers), will include members representing relevant patient organisations and parent support groups and parent ambassadors with relevant experiences. The overarching goal of the PPI Advisory Group will be to ensure the preferences and priorities of parents are central to the research and are included through every stage. The PPI Advisory Group will meet every quarter to review study materials and approaches to recruitment and help interpret findings.
By having a separate PPI Advisory Group we hope to create a meeting space where parents can feel comfortable to share their views. During the study the PPI members of the Study Advisory Group and the PPI Advisory Group will be supported through one-to-one calls or emails to check-in with members after meetings. These interactions support members and also allow them to share or discuss items they didn’t feel comfortable bringing up in the group. Informal training will also be available, for example to explain newborn screening and the Generation Study.
STUDY OVERVIEW
Key: GS = Generation Study
RATIONALE
The Generation Study aims to explore the benefits, challenges, and practicalities of offering WGS to parents of newborns, to accelerate diagnosis and access to treatment for rare genetic conditions. The Process and Impact Evaluation has been funded by Genomics England to provide an independent mixed-methods evaluation of the use of gNBS for the early diagnosis of rare, childhood-onset, actionable genetic conditions. The evaluation will examine whether offering gNBS in routine care is feasible and acceptable and will inform our understanding of the clinical utility and cost effectiveness of offering gNBS. We will also explore the attitudes and experiences of parents who have been offered gNBS and consider the clinical and psychosocial impacts, both positive and negative.
THEORETICAL FRAMEWORK
The evaluation will draw on several theoretical frameworks and measures. The use of theory-based and pre-specified constructs will help to generalise the findings and enable integration across the various studies, enabling us to build a stronger evidence base. For example, in order to understand the perspective of parents, we will use a number of parent reported outcome measures including decisional regret, and quality of life in our survey.
The philosophical approach that will underpin our mixed methods study design is pragmatism.1, 2 Pragmatism is a philosophical movement that states that an ideology or approach is true if it works satisfactorily. In that sense at its core it is problem-centred and puts the research question and generation of useful, actionable results above all else.2 It is frequently used in mixed methods research because it is focused on real-world issues and problems that need solving and has no loyalty to a specific philosophical stance. In that sense it permits different assumptions about worldviews and values both subjective and objective knowledge. Importantly, it allows the researchers to respond to the needs of the study as new evidence comes up, such as making methodological adjustments e.g. to research questions and/or data collection methods which will clearly be important given the fast turnaround of the project.
For the Process Evaluation we will use a theory driven approach to understand how the Generation Study is being implemented. We will draw on both the Consolidated Framework for Implementation Research (CFIR)3 and Normalisation Process Theory (NPT)4–6 when developing the interview topic guides and analysing the interview data. The CFIR can be used as an explanatory framework to systematically assess the contextual factors including barriers and facilitators that influence implementation and adoption,3 and has been used previously to evaluate implementation in other areas of genomic medicine.7–10 This framework is also well-suited to approaches involving rapid-cycle evaluation.11 The CFIR framework provides a taxonomy of operationally defined constructs that are likely to influence implementation of complex programs, organised into five major domains: 1) Intervention Characteristics (features of the intervention itself which might influence implementation e.g. complexity); 2) Outer Setting (features of the implementation organisation e.g. leadership engagement); 3) Inner Setting (features of the external context or environment e.g. readiness); 4) Characteristics of Individuals (e.g. knowledge and beliefs of individuals); and 5) Process (strategies or tactics which might influence implantation e.g. planning).
In addition, as this is the first time gNBS will be implemented within a public health setting on a large scale we will also draw on NPT, which is an implementation process theory that focuses on the work that individuals and groups do to enable an intervention to become routine or normalised.4–6 NPT emphasises agency, cooperation and coordination in a social system as key components of implementing complex interventions. NPT can be applied to explain how changes in the way people think about and use the intervention occur and identify factors that promote or inhibit the routine incorporation of the intervention into everyday practice. NPT proposes four constructs that represent different kinds of work that individuals and groups do around implementing a new practice: Coherence, Cognitive Participation, Collective Action, and Reflexive Monitoring. NPT will be used to guide the development of survey and interview questions and to inform the analysis of qualitative interview data.
STUDY DESIGN
This study will use a mixed methods research design using qualitative and quantitative approaches for data collection. Mixed methods approaches are frequently used to characterise complex healthcare systems as comparing the results of multiple data sets provides a more complete understanding of the topic.12 To deliver this evaluation within the 18 month time frame with formative feedback, we will draw on the rapid feedback and rapid cycle evaluations used in the Rapid Research Evaluation And Appraisal Lab (RREAL).13, 14
Our evaluation comprises an initial scoping study and seven evaluation studies:
Scoping study: A scoping study will be carried out during the first months of the evaluation to determine the scope of the evaluation and engage with stakeholders.
Study 1: Identifying goals, challenges and early lessons in implementation: Interviews with professionals who are the Generation Study’s “designers” and “early implementors”; shadowing of processes and pathways; and documentary analysis.
Study 2: Impact, experiences and attitudes of parents: Survey and follow-up interviews with parents following receipt of gNBS results to assess acceptability, experience, attitudes and impact (positive and negative).
Study 3: Gathering wider professional viewpoints about the Generation Study and gNBS implementation: Survey with staff delivering the Generation Study at early adopter sites and professionals from a range of relevant backgrounds across England.
Study 4: Views of the rare disease community: Surveys and interviews with advocates from rare disease support groups to examine attitudes to gNBS including positive and negative impacts as well as unintended consequences on parents and the rare disease community more broadly.
Study 5: Public views: Survey conducted with a market research company to capture views of with a diverse group of parents in terms of age, gender, ethnicity, socio-economic status and geographical location to evaluate the key findings around acceptability and attitudes towards gNBS with parents who have not been involved in the Generation Study.
Study 6: Cost effectiveness evaluation: Assessment of the impact of gNBS on healthcare resource use and costs, quality of life outcomes and non-health-related outcomes, to inform the Genomics England health economics model.
Study 7: Clinical utility assessment: Assessment of the clinical utility of gNBS for health-related outcomes by comparing hospital contact and mortality rates in various groups of children (‘condition suspected’ and false positives, population controls), making use of the longitudinal routinely collected healthcare data linked to the Generation Study and control groups primarily drawn from ECHILD.
RESEARCH PLAN
Scoping study
A scoping study will be carried out to determine the scope of the evaluation, identify and engage with key stakeholders and agree the final evaluation and dissemination plan with Genomics England. Discussions around the scope will include, prioritising the research questions set out by Genomics England, evaluation methods, timeline and dissemination plan. The scoping study will include meetings with key stakeholders and a review of relevant documents to understand the context of the Generation Study. We will also review the published and grey literature to identify any accessible study materials, (e.g. surveys and topic guides) being used in international evaluations of gNBS that could inform the development of our own study materials.
Study 1: Identifying goals, challenges, and early lessons in implementation
Research questions
Is the use of WGS as a tool for early diagnosis of rare, childhood-onset, actionable genetic conditions feasible and acceptable to professionals?
What are the implementation processes being used to deliver the Generation Study and how do these vary across sites?
What are the effective approaches to implementation for delivery of gNBS including identified barriers and facilitators?
What are professionals’ experiences of and attitudes towards delivering gNBS?
What are professionals’ education and training needs to support delivery of gNBS?
What factors impact access and uptake to the Generation Study?
Study design
The aim of this study is to understand the goals, challenges and early lessons of implementing gNBS in the Generation Study. This will include exploring the programme theory underpinning the Generation Study, assessing acceptability, feasibility, barriers and facilitators of implementation and impacts on staff time.
This will be done using four approaches;
Qualitative interviews with 15-20 key stakeholders involved in designing, planning and discussing the Generation Study (“designers”); including professionals from organisations such as Genomics England, NHS England, clinicians from a range of specialties (both mainstream and genetics), policy makers and commissioners.
Qualitative interviews with 5-6 professionals at each of the first 5-6 NHS Trusts undertaking recruitment to the Generation Study (“early implementers”); including clinicians from a range of specialties who are involved with recruitment and consent, returning results and study planning and management and staff education (e.g., midwives, genomics associates neonatal, paediatric and genetic specialists, researchers and data administrators and laboratory staff).
Documentary analysis of relevant documents such as reports, policy documents, journal articles and meeting presentations.
Shadowing of recruitment, consent and sampling processes at each of the first 5-6 NHS Trusts undertaking recruitment to the Generation Study.
Interview topic guides
Interview question guides have been developed with consideration for the existing literature, relevant theoretical frameworks (CFIR3 and NPT4–6) and the academic, clinical and PPI expertise of our research team. The topic guides will be continuously revised to address any new topics that emerge during the interviews. For the interviews with “designers”, the topic guide includes: Design of the Generation Study; Feasibility of implementing the Generation Study; Impact of the Generation Study; Barriers and facilitators; and Personal experience and reflections. For the interviews with “early implementors”, the topic guide includes: Design of the Generation Study; Acceptability of the Generation Study; Feasibility of implementing the Generation Study; Impact of the Generation Study; Barriers and facilitators; and Personal experience and reflections.
Data collection
Interviews will be carried out by phone, video call or face-to-face at the professional’s place of work. The interviews are expected to last approximately 45 to 60 minutes. Interviews will be conducted by a researcher experienced in qualitative interviews. Interviews will be digitally recorded, professionally transcribed verbatim and then anonymised.
Participant and non-participant observations will be carried out approximately 2-6 months after recruitment commences at each of the first 5-6 recruitment sites for the Generation Study. We will use a shadowing approach to observe and map how the Generation Study is delivered at each site including; how parents are identified and invited to take part in the Generation Study, length and content of consent discussions, processes for sample collection and dispatch, and which staff are involved.15, 16 Participant and non-participant observations will be carried out in the clinical areas where the discussions and tasks related to the Generation Study take place. Relevant staff meetings for observation will be identified by discussion with the Generation Study team at each site. These may include Generation Study team meetings, departmental meetings, multi-disciplinary team (MDT) meetings or educational events. The discussions between staff and parents will also be observed, this may include any initial discussions introducing the Generation Study to parents and taking consent to participate. A structured observation guide will be used to record all field notes to ensure consistency in the collection of data across researchers and sites. The field notes will record high-level summary information only. No identifying or confidential information about individuals will be recorded. This guide will be iteratively updated to explore themes emerging from the data.
To help us understand the reasons parents decline to take part in the Generation Study, a very brief ‘exit survey’ will be undertaken with those parents who decline when the Generation Study is discussed with them. This will be done by the health professional who is inviting the parents to take part in the Generation Study. Parents will be asked their reason(s) for declining to take part and if they choose to give a reason the health professional will record the response on a form that will comprise a short list of common reasons to decline and an open-ended question capture additional reasons.
Documentary evidence such as policy documents, journal articles and meeting presentations will be collated for analysis.
Sample size
We will conduct interviews with the “designers” and “early implementors” of the Generation Study until saturation is reached. We anticipate conducting 15-20 interviews with “designers” and 25-30 interviews with the “early implementers” (5-6 from each site) based on previous similar research.17, 18
Observations will be conducted at each of the first 5-6 recruitment sites for the Generation Study. As part of the observations at each site, we will observe 4-5 discussions between staff and parents.
The exit survey will be conducted at each of the first 5-6 recruitment sites for the Generation Study. Professionals discussing the Generation Study with parents will be asked to complete the exit survey with any parents who decline within a 3 month period.
Recruitment
Interviews: The designers and early implementers of the Generation Study will be purposively sampled and invited to take part in a telephone or video interview. Relevant professionals will be identified by the research team with input from the Generation Study PIs at recruiting sites.
“Snowball sampling” will also be used to ensure that key professionals not known to the team are invited. Professionals will be invited to take part in a semi-structured interview. Potential participants will be sent an invitation email along with a Participant Information Sheet describing the purpose of the interviews. The professionals will be asked to contact the research team via telephone or email if they are interested in participating in an interview. Prior to the start of the interview the participant will be asked if they have any questions, and it will be explained that the discussion will be recorded and transcribed but that no identifying information will be included on the transcript. Participants will then be asked to read and sign the consent form (or to give verbal consent (audio-recorded) for an interview conducted by telephone or video call).
Observations: Observations will be conducted at each of the first 5-6 recruitment sites for the Generation Study. We will adopt a ‘shadowing approach’ whereby a researcher shadows members of staff who are discussing and recruiting parents into the Generation Study. Observations will include the discussions professionals have with parents when first introducing the Generation Study, inviting them to take part and taking consent. These discussions may take place in the hospital or via phone or video call and this may differ between sites, depending on the individual processes developed for delivering the Generation Study at each site.
Prior to any observations taking place that involve both staff and parents, we will share a Participant Information Sheet with the professional and give them the opportunity to discuss the study with the researcher. The Participant Information Sheet will explain the purpose of the observation, the type of data to be collected (e.g. length of the discussion, topics discussed etc) and that the data collected will be anonymous. In the Participant Information Sheet and in the initial discussion with the researcher, it will be made clear that the health professional is required to gain verbal consent from parents for a researcher to observe the discussion. It will also be emphasised that the researchers should not have access to any confidential information about the parents prior to the parents giving consent for the researcher to observe the discussion. A guidance document to support health professionals to explain the purpose of the observation, the type of information collected and how to take verbal consent from the parents will be shared.
Observations of meetings: Before attending any specific staff meetings, the researcher will contact and seek permission to attend from the meeting Chair. A Participant Information Sheet for meeting attendees will be circulated to all attendees with the meeting papers; this will re-iterate the study details and will give an opportunity for any members of staff attending the meeting to opt out of the observation and not have their contributions to the meeting recorded in the field notes made by the researcher. The Participant Information Sheet will also be available at the beginning of the meeting. All field notes will be high level and no identifying or confidential details about individuals will be recorded.
Exit Survey: Professionals discussing the Generation Study with parents will be asked to complete the exit survey with any parents who decline to take part in the Generation Study within a 3 month period.
Data analysis
We will use both rapid qualitative data analysis and in-depth analysis.
Rapid qualitative data analysis
Analysis will be carried out in parallel with data collection and facilitated through interview notes and RREAL sheets; a working document where high level data are organised into categories.19 During the interviews and observations, the researchers will take notes that will be summarised and organised in the RREAL sheet immediately after each data collection episode. The RREAL sheets can then be used to summarise and share emerging findings on an ongoing basis.19 The RREAL sheets will also be used to identify topics that will need to be explored further using in-depth analysis.
In-depth analysis
Data from interviews, the exit survey and documents will be analysed using framework analysis.20 This is an approach that facilitates identification of key themes as well as commonalities and differences in the data through comparison within and across cases. The same framework will be applied to the analysis of both sets of interviews as well as the exit survey and documentary evidence to enable cross-referencing and comparisons across the data using a coding matrix in Excel.
Study 2: Impact, experiences and attitudes of parents
Research questions
Is the use of WGS as a tool for early diagnosis of rare, childhood-onset, actionable genetic conditions feasible and acceptable to parents?
What are parents’ experience of joining the Generation Study?
Did parents perceive that information about gNBS was effectively conveyed and that they were able to make an informed choice?
What factors impact access and uptake to gNBS?
What are parents’ information and support needs to facilitate decision making, receiving results and ongoing care and support networks?
Can findings be returned effectively using the approach adopted for the study overall?
What are parents’ experiences of receiving a ‘no condition suspected’ gNBS result?
What are parents’ experiences of receiving a ‘condition suspected’ and a ‘confirmed diagnosis’ gNBS result, including the downstream pathways for parents receiving those results?
What are the wider positive and negative impacts on families including parental mental health?
Study design
We will conduct a mixed-methods study including a survey and a sub-set of follow-up interviews with parents following receipt of gNBS results to assess acceptability, experience, attitudes, quality of life and impact (positive and negative). The survey will be disseminated after parents receive gNBS results (including ‘no condition suspected’, ‘condition suspected’ and uncertain results).
Survey content
The survey will include a mix of bespoke questions and validated measures and will be informed by our study and PPI advisory groups. The survey will follow on from the baseline survey conducted by Genomics England to explore parents’ experience of the consent process, which will include, amongst other questions, the EQ-5D-5L21 to assess parental quality of life, the GAD-7 anxiety measure22 and the SURE measure of decisional conflict.23 In order to measure quality of life and non-health outcomes after receipt of results, we are considering the following measures for inclusion in the survey: parental quality of life, e.g. the EQ-5D-5L21 and/or PedsQL Family Impact survey24; child quality of life (EQ-TIPS 25 or PedsQL-I26 which are specifically for newborns); and non-health outcomes including psychosocial impact of receiving genomic findings (e.g. FACToR27 or PAGIS28), parental personal utility (e.g. PrU,29 GOS30); perceived uncertainties associated with genome sequencing results (e.g. PUGS31); parent-child bonding (e.g. Mother to Infant Boding Scale);32 parents’ perception of child’s vulnerability (e.g. The Vulnerability Baby Scale);33 parent relationship (e.g. Kansas Marital Satisfaction Scale);34 anxiety (e.g. Edinburgh Postnatal Depression Scale35); decisional regret (e.g. Decisional Regret scale36); knowledge and attitudes towards gNBS including perceived benefits and risks and motivations for taking part;37 family experience of joining the programme including satisfaction with receipt of results;38 signposting to support groups and impact on wider family e.g. key decision-makers, cascade testing. We will also ask parents to recall any out-of-pocket costs incurred from consent to results receipt.
Interview topic guides
Interviews will be conducted with a subset of survey responders who have indicated on the survey that they are willing to take part in an interview, expected to last 30-45 minutes. We will purposively select participants to ensure a range in terms of participant characteristics and gNBS results. The interview topic-guide will be co-designed with the study and PPI advisory groups. Interview topics are likely to include: acceptability of gNBS including potential roll-out as a service, perceived benefits and concerns, experience of joining the Generation Study (satisfaction with the consent process, information and support needs, timing of approach, turnaround time, receiving results, ongoing care and support networks), and positive and negative impact of receiving gNBS results including psychological impact, any unanticipated outcomes and perceived utility e.g., for reproductive decision-making, other family members.
Data collection
Surveys can either be completed on paper and sent back using a freepost envelope, by phone with a researcher or online using REDCap (a link and QR code will be included in the Participant Information Sheet). The survey will take 10-15 minutes to complete.
Interviews will be carried out by phone or video call. The interviews are expected to last approximately 45 to 60 minutes. Interviews will be conducted by a researcher experienced in qualitative interviews. Interviews will be digitally recorded, professionally transcribed verbatim and then anonymised. If participants prefer to not have the interview recorded, then the interviewer will take written notes during the interview.
Sample size
For the survey, we have estimated a sample size of 500 completed surveys from across all early adopter sites. This is based on our previous survey study where we recruited 504 participants taking part in the 100,000 Genomes Project across 6 sites in 15 months.39 A sample size of 500 would allow for a difference in the mean score of 5 between those receiving a ‘condition suspected’ and those receiving a ‘no condition suspected’ gNBS result on the decisional regret scale (at 80% power, α = .05). We will aim to include a subset of responders who receive a ‘false positive’ result.
We anticipate interviewing ∼50 parents so that we can include parents from a broad range of participant characteristics (10% of survey responders).
Sampling and recruitment
Survey participants will be a sub-set of participants recruited into the Generation Study. We will recruit participants across demographic groups e.g. ethnicity, education, health literacy, gender using a stratified random sampling approach to explore the role of health inequalities. Genomics England will send an invitation to take part in the study by email to the potential participants. The email will include a Participant Information Sheet describing the survey and interview study and a link to complete the survey online. Potential participants will be able to request a paper copy of the survey that can be returned by reply-paid post. Recruitment will continue until the minimum number of completed surveys have been collected. All participants who complete a survey will be sent a £10 Amazon voucher in appreciation of their time.
A subset of survey responders will be purposively sampled to take part in an interview, aiming for a range in terms of gender, age, ethnicity, education, health literacy, place of enrolment, parity, type of gNBS results received. Participants who take part in an interview will be sent a £20 Amazon voucher in appreciation of their time.
The contact details of the research team will be included in the first page of the survey, so that potential participants can contact the research team with any queries. If participants do feel distressed by the survey or interview, the participant information sheet contains several alternatives for seeking advice or support including, a counsellor from the organisation rareminds (https://www.rareminds.org/) who provide mental health support for the rare disease community and are independent of the Generation Study, their regional clinical genomics service, their general practitioner (GP) or their Trust Chaplains.
Data analysis
The survey will be analysed using both descriptive and inferential statistics to gain a nuanced understanding of acceptability and uptake amongst different population groups. For the analysis of the interviews will use rapid qualitative data analysis so that results can be fed back quickly, which will be followed by in-depth analysis.
Rapid qualitative data analysis: As described for Study-1, rapid qualitative data analysis will be carried out in parallel with data collection and facilitated through interview notes and RREAL sheets.19 The RREAL sheets will be used to summarise and share emerging findings on an ongoing basis.19 The RREAL sheets will also be used to identify topics that will need to be explored further using in-depth analysis.
In-depth analysis
Data from the interviews will be analysed using thematic analysis.40 Analysis will involve an iterative process where data are coded, compared, contrasted and refined to generate themes. Analysis will be conducted by at least two researchers to provide rigour. Coding will be informed by the topic guide (deductive analysis) and may also include new topics or unexpected findings (inductive analysis).41 Data analysis will be facilitated by using a coding matrix in Excel and, as needed, NVivo version 13 (QSR International, Pty Ltd).
Linkage to the Generation Study data
Each parent invited to complete a survey will be assigned a unique code by Genomics England to facilitate tracking and linkage. The survey data will be stored on and analysed using the UCL Data Safe Haven. This data-set will also be shared securely with Genomics England and the unique code will be used to link an individual’s survey data with the Generation Study data including diagnostic outcomes data and longitudinal secondary datasets within the secure environment of the National Genomic Research Library (NGRL). Only relevant members of the Generation Study and Evaluation Study team’s will be able to access these linked data sets, which will be stored within a ring-fenced area of Genomics England’s secure NGRL, which will not be accessible to other researchers. Any practices in relation to the NGRL will be carried out in line with Genomics England’s NGRL approved protocol.42
Study 3: Gathering wider professional viewpoints about the Generation Study and gNBS implementation
Research questions
Is the use of gNBS as a tool for early diagnosis of rare, childhood-onset, actionable genetic conditions feasible and acceptable to the wider workforce?
What are wider workforces’ experiences and attitudes to the use of gNBS as a tool for early diagnosis of rare, childhood-onset, actionable genetic conditions?
What are the downstream implications of offering gNBS including impacts on the wider clinical workforce?
Study design
The aim of this study is to obtain a broader understanding of professional views and experiences of offering gNBS in the Generation Study and gather wider opinions around introducing gNBS into clinical practice as part of routine care. To do this we will develop an anonymous cross-sectional survey that will be circulated after the Generation study has been scaled-up for implementation at early adopter sites. Potential participants will include: A. professionals delivering the Generation Study at all NHS Trusts that are actively recruiting, B. professionals from the NHS Trusts where babies with a condition suspected result from gNBS have been referred for return of results and clinical care, and C. professionals from a range of relevant backgrounds from across England who are not directly involved with delivery of the Generation Study.
Survey content
Survey development will be informed by the existing literature, findings from the qualitative interviews with professionals in Study 1 and the expertise of our advisory groups. Questions will include a Knowledge, Attitudes and Practice (KAP) survey, which uses a mix of closed and open ended questions to elicit what is known (knowledge), believed (attitude), and done (practiced) for a particular topic.43 KAP surveys are widely used and can identify misunderstandings, misconceptions, as well as facilitators and barriers to implementation. The survey will also include validated measures that address the concepts of acceptability and feasibility.44 The survey will capture differences or similarities in processes used by Generation Study recruitment sites for discussing gNBS, obtaining samples and reporting results. We will also invite respondents to identify anticipated challenges for delivering gNBS in routine care, expected workforce implications, potential impacts on healthcare systems, education needs and overall views. Responses will be anonymous, with only the specification of the NHS Trust requested. Survey development and validation will include piloting with 5-10 professionals from different backgrounds to ensure consistency and clarity.
Data collection
The survey will be hosted online using REDCap and will be made available for 6 weeks. Completion of the survey is expected to take approximately 20-25 minutes per participant.
Sample size
Participants will be purposefully sampled to ensure there is maximum variation professional background and geographical location. A sample size of 200, including 5-10 participants from each Generation Study recruitment site and 10-15 from each of the seven GLH/GMSA regions in England has been estimated on the basis of our previous research experience as being sufficient to provide a depth and breadth of opinions and attitudes. Due to the nature of the survey, it is not proposed that this sample will be statistically representative of the population groups consulted.
Sampling and recruitment
Recruitment will involve two approaches. 1. The Generation Study PI(s) at each recruitment site will be asked to provide a list of potential participants that will include professionals involved in recruitment and consent for the Generation Study, professionals from the NHS Trusts where babies with a positive result from gNBS have been referred for return of results and clinical care and professionals involved in local care pathways relevant to newborn screening. The research team will circulate an invitation email with a link to the online survey to the potential participants. Three reminder emails will be sent. 2. A study flyer with a link to the survey will be circulated through social media and email lists of professional bodies such as the Royal College of Midwives, the Royal College of Paediatrics and Child Health, the Royal College of Nursing and the British Society for Genetic Medicine.
Data analysis
Quantitative data from the survey will be analysed using descriptive statistics and open-ended survey questions will be analysed using thematic analysis.40
Study 4: Views of the rare disease community
Research questions
How prepared are patient organisations to support families taking part in gNBS?
What impact do patient organisations think gNBS will have on the families they support now and in the future?
Study design
Many rare disease patient advocacy organisations provide direct support to affected individuals e.g. through helplines, as well as working to influence the services and support available through statutory services. Many are small organisations with few staff, sometimes working as volunteers. To examine views on gNBS from the perspective of these organisations and the rare disease community more broadly, we will conduct surveys and interviews with advocates from rare disease support groups.
Survey content
An online survey will be co-produced with our PPI Advisory Group and representatives of a small number of patient advocacy organisations. Questions to be explored will include: Do rare disease advocacy organisations have concerns about the Generation Study and gNBS generally in terms of its impact on them or the people they support? Were they able to take part in consultations run by Genomics England before the study was launched, and what are their views of these consultations? Are there changes in demand for support and are they prepared (greater volume/new situations e.g. pre-symptomatic diagnoses)? How are parents finding them (e.g. signposted from clinical services)? What do they see as benefits for their community? Do they foresee any unintended consequences?
Interview topic guides
Interviews will be conducted with a subset of survey responders. The interview topic-guide will be co-produced with the PPI Advisory Group. Interview topics will reflect the survey content, providing an opportunity to discuss these issues in more depth.
Data collection
The survey will be hosted online through REDCap and will be made available for 6 weeks. Completion of the survey is expected to take approximately 20-25 minutes per participant. Interviews will be carried out by phone or video call. The interviews are expected to last approximately 45 to 60 minutes. Interviews will be conducted by a researcher experienced in qualitative interviews. Interviews will be digitally recorded, professionally transcribed verbatim and then anonymised. If participants prefer not to have the interview recorded then the interviewer will take written notes during the interview.
Sample size
There are over 200 organisations in the UK who provide support to parents and families impacted by rare diseases. Based on previous surveys with these groups, we anticipate around 50% of the organisations will respond, giving us a sample size of 100 (Amy Hunter, personal communication), which will be sufficient to provide a depth and breadth of opinions and attitudes. Follow-up qualitative interviews will be arranged with ∼10 representatives of rare disease organisations (10%).
Sampling and recruitment
An invitation email that will include the Participant Information Sheet and a link to the online survey will be circulated by the research team at Genetic Alliance UK to the ∼230 member organisations of Genetic Alliance UK, and the organisational members of The Neurological Alliance who encompass rare conditions. Three reminder emails will be sent during the six weeks the survey will be open.
At the end of the survey respondents will be asked if they would be interested in taking part in a follow-up interview. Survey participants invited to take part in a follow-up interview will be purposively sampled to ensure we gather more in-depth views from organisations from a range of sizes and representing a range of different types of conditions.
Data analysis
Descriptive statistics will be generated from the survey data. Open text responses to the survey and qualitative interview transcripts will be analysed using thematic analysis.40
Study 5: Public views
Research questions
What are parents from the general public (i.e. those who have not been invited to take part in the Generation Study) attitudes towards gNBS?
Do parents from different ethnic or educational backgrounds differ on their views of gNBS?
Do parents from the general public view gNBS as acceptable?
Study design
We will conduct a survey with parents from the public who have given birth in the previous two years and who have not been invited to take part in the Generation Study. The survey will ascertain the views of parents towards gNBS and in particular to examine whether the acceptability and attitudes identified through our survey and interviews with parents in Study 2 are shared by the public. The survey will be conducted with the market research company Dynata (https://www.dynata.com/), who we have worked with previously.45 A service agreement will be in place between Dynata and UCLC.
Survey content
An online survey will be developed to assess acceptability, views and attitudes towards gNBS. Survey development will be informed by the findings from the surveys and interviews with parents taking part in the Generation study (Study 2) and the expertise of our Study and PPI advisory groups.
Data collection
We will conduct an online survey that will be set up on REDCap through the market research company Dynata.
Sample size
As the survey will be conducted through a market research company, we will request quotas for specific population groups, in particular, those whom we may not have reached through our parent survey. We will recruit 200-250 survey responders to enable sufficient numbers to compare across groups. An a priori power analysis was conducted to determine the minimum sample size required to test the study hypothesis. The required sample size to achieve 80% power for detecting a medium effect, at a significance criterion of α = .05 was n = 216 for tests of comparisons across sub-groups.
Recruitment
Recruitment will be done by the market research company Dynata. An invitation to complete an online survey will be circulated by Dynata to women and men (over the age of 18) who have had a child in the last two years. This will ensure that we capture the views of parents who have recently had a child and have been through antenatal and neonatal services.
In the first instance, Dynata, will send the survey link to a small group of potential participants as a pilot. The aim of the pilot will be to test out the recruitment method and to ensure the survey is completed as intended. We will aim to collect 20 completed surveys in the pilot study. Following the pilot, the survey will be sent to a larger number of potential participants and recruitment will continue until the target sample size is reached across all requested population groups. Dynata will send an invitation to participate with a link to complete the survey by email or potential participants will access Dynata’s system themselves to look for surveys. Participants who complete the survey will be paid Dynata’s standard E-reward payment that is equivalent to £1.00 for a 20 minute survey.
The contact details of the research team will be included in the first page of the survey, so that potential participants can contact the research team with any queries. To make sure parents also have a point of contact for support if the survey content raises any worries or questions, at the end of the survey we have provided advice for seeking support from their GP or a counsellor from the organisation rareminds (https://www.rareminds.org/) who provide mental health support for the rare disease community and are independent of the Generation Study.
Data analysis
Survey data will be analysed using descriptive and inferential statistics.
Study 6: Cost effectiveness evaluation
Research questions
What is the impact on secondary healthcare resource use including estimates of end-to-end costs – from recruitment, through sample processing, sequencing and interpretation through to clinical care?
What are the positive and negative impacts on health-related outcomes including: morbidity, mortality and quality of life?
What is the impact on non-health-related outcomes including: perceived personal utility and psychosocial wellbeing?
Study design
We will evaluate the impact of gNBS on healthcare resource use and costs, quality of life outcomes and non-health-related outcomes. Parameter estimates will be generated to support Genomics England with the development of an economic model to answer the primary cost-effectiveness research question. Specifically, we will:
1. Estimate the costs associated with sequencing newborns in the Generation Study, and the costs of implementation and scaling up;
2. Estimate the impact of gNBS on healthcare resource use and associated costs, and on out of pocket costs for children and their families, compared to children who do not undergo sequencing;
3. Estimate the impact of gNBS on morbidity and quality of life outcomes in children, and quality of life outcomes in parents, compared to children who do not undergo sequencing and their parents;
4. Estimate the impact of gNBS on non-health-outcomes for parents, compared to parents of children who do not undergo sequencing.
Data collection and sampling
A microcosting study will be conducted alongside Study 2 to estimate the costs associated with sequencing children, from sample collection to return of results. Current screening costs will be extracted from the literature.46, 47 Data on the costs of implementation and scaling up will be collected in the interviews (Study 1) and cross-sectional survey (Study 3) with professionals and combined with data on likely uptake (Study 2).
The analysis of secondary care resource use for children undergoing gNBS will use the linked Hospital Episode Statistics data made available by Genomics England, combined with NHS Reference Costs. Secondary care resource use and costs for participants in the 100,000 Genomes Project, and for ‘no condition suspected’ participants in the Generation Study, will be used as controls. Data on out of pocket costs accrued by participants in the Generation Study (e.g. over-the-counter medications, caregiver time) will be collected via WS1 of the Impact Evaluation, following receipt of results. Comparator costs will be extracted from the literature.48, 49 Analysis of morbidity and mortality outcomes will use data made available in the Generation Study for newborns undergoing sequencing, and for age matched children with comparable diagnoses who have not undergone sequencing.
Data on quality of life outcomes in newborns will be collected using proxy-completed age-specific instruments (e.g. EQ-TIPS, PedsQL), administered in the parent survey (Study 2).
Baseline values are available in the literature50 and from the instrument developers. Data on the quality of life of parents will be collected using multiple instruments (e.g. EQ-5D-5L, PedsQL Parent Family Impact survey), via the baseline consent survey distributed by Genomics England, and in the aforementioned parent survey. Quality of life data for children not undergoing sequencing, and their parents, will be estimated using established methods.46 Non-health outcomes will be measured using multiple instruments (e.g. FACToR, PAGIS, PrU, GOS, Edinburgh Postnatal Depression Scale). Data will be collected in the parent survey (Study 2) and supplemented with data from other newborn sequencing studies applying these instruments (e.g. BabySeq 2), via established collaborations.
Data analysis
For all parameters, descriptive statistics will be presented (e.g. mean values, standard deviations, confidence intervals). Summary statistics for cost parameters will be expressed across specific time periods (e.g. per month/year) to align with the economic model. Results for quality of life instruments will be presented at the domain level and, where value sets exist, as utility scores. Summary statistics will be presented for predefined subgroups (e.g. by sequencing result, by condition). Differences will be calculated between parameter values at baseline and after the return of results. If required for the economic model, we will use quasi-experimental approaches (e.g. difference-in-difference) to further quantify differences in outcomes between sequenced and non-sequenced children. Threshold analyses will explore the impact of key assumptions. Missing data will be quantified as required, and analyses adjusted using appropriate methods (e.g. multiple imputation).
Study 7: Clinical utility assessment
Research questions
What are the positive and negative impacts on health-related outcomes (morbidity, mortality)?
What is the prevalence of the conditions looked for in the newborn population?
How many are ‘confirmed diagnosis’ and how many would be expected if it were adopted nationally?
What is the level of uptake of the programme and what would the level of uptake of genomic newborn screening be if it were adopted nationally?
Study design
We will use a series of cohort studies to assess the impact of gNBS on health-related outcomes. Our objectives are to:
1) Estimate hospital contact & mortality rates among children who are confirmed to have a rare condition via gNBS, compared to children with similar conditions who were diagnosed through routine clinical practice
2) Estimate hospital contact & and mortality rates among children who are ‘false positive’, compared to a) children in the Generation Study who receive a ‘no condition suspected’ result, and b) the general population of children in England
3) Estimate the impact of introducing the programme in England on health service use and mortality, taking into account differential selection into the Generation Study
Data sources
We will utilise longitudinal healthcare data that are being linked to the Generation Study (including HES, ONS mortality data, the NNRD). We will work with Genomics England to determine the optimal datasets for control groups, however we propose to draw comparator data from the Education and Health Education and Child Health Insights from Linked Data (ECHILD). (https://www.ucl.ac.uk/child-health/research/population-policy-and-practice-research-and-teaching-department/cenb-clinical-20) ECHILD links administrative health data, including HES, to education data for all individuals born in England since 1997. A mother-baby link is being incorporated into ECHILD, allowing future studies of the health of mothers and siblings of children with rare conditions.
Data analysis
The primary outcome will be emergency hospital contact (accident and emergency [A&E] attendances and admission rates, derived from HES. Our secondary outcomes will be NICU admission rates (from NNRD), planned hospital admission rates and mortality. Our primary focus is on secondary care use; we expect a small number of deaths in the recruited cohort. Given the start date of recruitment, the oldest children in the Generation Study will be over 12 months of age at the start of the study. Early development and education outcomes will therefore be out of scope, however we will develop analysis approaches and code to be applied in future natural history studies using Community Services Dataset and National Pupil Data (part of ECHILD).
Objective 1: We will use linked Generation Study-HES-mortality data from the NGRL to calculate A&E attendance, emergency hospital admission and mortality rates among true positive children. We expect to group children with similar conditions for analyses; useful groups will be agreed with the Genomics England Team. We will use the linked Generation Study-HES data for true positive children to define these conditions in HES using clinical code lists. We will apply these definitions in ECHILD and estimate prevalence of these conditions in general population, hospital use and mortality among children with similar conditions diagnosed via routine practice. This approach can be extended to examine early development (via CSDS) and National Pupil data.
Objective 2: We will estimate hospital admission and mortality rates among ‘false positive’ and ‘no condition suspected’ children using the linked Generation Study-HES-mortality data. Several population control groups can be derived using ECHILD, including all children born in England during particular time periods, children meeting inclusion criteria born in participating trusts during non-recruiting periods, or children meeting inclusion criteria born during recruiting periods but in non-participating trusts – to be agreed with Genomics England. For both objective 1 and 2, we will use generalised linear models and/or survival analyses to examine impact of gNBS on hospital admission and mortality rates.
Objective 3: We will use results from objective 2 and 3 applied to ECHILD data to estimate the number of hospital admissions and deaths prevented by gNBS across England. We will also use ECHILD data to estimate level of gNBS uptake and screen positives if the programme was applied nationally. We will use inverse probability weighting and similar methods to account for likely under-representation of some groups of children consented to the Generation Study (such as children from minority ethnic groups and more deprived families), and the exclusion of premature babies, to examine gNBS’s impact on health inequalities.
Integration of findings
Research findings from all studies will be integrated using a mixed-methods coding matrix that will be linked to each research question. Drawing together the findings of the individual studies will generate a holistic picture of the outcomes of our Process and Impact Evaluation.
SUMMARY OF TIMELINES
PROJECT MANAGEMENT
Team working
The full research team will meet at fortnightly ‘check-in’ meetings (videoconference) to monitor overall progress against the project plan. At these meetings the lead research for each study will be asked to feedback progress against their deliverable(s). Any potential issues or delays will be discussed amongst the team, including whether wider discussion with the advisory teams is necessary. Smaller working groups will be established to collaborate on individual studies. UCL Consultants Ltd (UCLC) will provide project management, financial monitoring and risk management support throughout the evaluation. UCLC will also take responsibility for contracts and account management.
Data sharing
Non-sensitive data such as data collection documents and study reports will be shared through the Office Teams platform which all members of the research team will have access to. De-identified research data such as anonymised survey data will be stored on the password protected UCL network. Personal data such as contact details DoB, ethnicity etc as well as survey responses, recordings of interviews and non-anonymised transcripts will be stored on the Data Safe Haven. Sensitive data will only be accessible to the research team through the Data Safe Haven. Any paper copies of surveys will be kept in a secure UCL office.
Collaboration with Genomics England
We will share formative feedback at a virtual meeting with Genomics England every 3 months to share our findings as they emerge. Findings will be shared via slide presentations with an agreed template. Formative feedback will include (1) sharing our evolving understanding of the programme theory and suggested refinements; (2) providing ‘real-time’ insights on the implementation of the Generation Study; (3) presenting staff views on the processes of implementation, and (4) discussing interpretation of findings. Formative reporting will also include an interim report delivered in month 9 and the final report delivered in month 24.
ETHICAL AND REGULATORY COMPLIANCE
This study will be conducted in line with the ethical framework set out by the NHS Health Research Authority and according to Good Clinical Practice (GCP) principles. The study design includes measures to safeguard the wellbeing and dignity of participants and all staff and procedures will comply with the Data Protection Act 1998 and the EU General Data Protection Regulation. Recruitment will only take place once Research Ethics Committee (REC) and Health Regulatory Authority (HRA) approvals are in place. The Generation Study and the Process and Impact Evaluation have been approved by the HRA and the East of England – Cambridge Central NHS REC (23/EE/0044).
RISK MANAGEMENT
UCLC has incorporated a risk management strategy and policy as part of its internal control and corporate governance arrangements. UCLC applies best practice in the identification, evaluation and cost effective control of risks to ensure that they are eliminated or mitigated. Risk considerations will be assessed throughout the project life cycle. UCLC will work with the research team and Genomics England to establish a Risk Register and appropriate escalation and risk management approaches at the project outset. The Risk Register will be a live document on a Teams site in which the project team can review the existing risks / add items at any time. The project manager from UCLC will be responsible for maintaining and updating the Risk Register, however, specific risks will be delegated to the relevant research team members. Updates to the Risk Register will be an agenda item at the research team’s fortnightly check-in meetings so that any identified risks can be discussed and addressed rapidly. Any potential issues or delays will be discussed amongst the team, including whether wider discussion with the advisory team or Genomics England is necessary.
DISSEMINATION
Our strategy for engagement, formative feedback and dissemination includes:
Peer reviewed scientific publications.
Presentations to scientific meetings, nationally and internationally.
Feedback of findings to professionals from a range of backgrounds.
Plain language summaries of findings, written with the help of the PPI Advisory Group, will be disseminated to parent and patient networks via meetings, newsletters and social media.
Authorship eligibility guidelines and any intended use of professional writers
Eligibility for authorship will follow the guidelines set out by the International Committee of Medical Journal Editors (ICMJE) (https://www.icmje.org/recommendations/browse/roles-and-responsibilities/defining-the-role-of-authors-and-contributors.html). No professional writers will be used.
Competing Interest Statement
James Buchanan has received travel support from Illumina and consulting fees from Genomics England.
Martin Vu has received a PhD scholarship from Illumina under the University of Melbourne/Illumina partnership.
Amy Hunter and Jennifer Jones are employees of Genetic Alliance UK. Genetic Alliance UK runs Rare Disease UK – a campaign for people with rare diseases and all who support them. Rare Disease UK receives financial support from a range of companies developing therapeutics for rare conditions.
All other authors have declared no competing interest.
Funding Statement
This manuscript presents independent research commissioned by Genomics England. The views expressed are those of the authors and not necessarily those of Genomics England, the NHS or the UK Department of Health.
Data Availability Statement
No data are associated with this manuscript as it is a study protocol.
Data Availability
No data are associated with this manuscript as it is a study protocol.
APPENDICIES
1. Questions in the Genomic England ITT not covered in the Process and Impact Evaluation that will be covered internally by Genomics England and/or through other collaborations
Feasibility and Acceptability
Whether approaches to implementation could scale
Can samples be taken consistently in a busy newborn setting and with sufficient quality to support WGS and analysis?
Can a sufficiently rapid end to end turnaround time be achieved from sample to issue of (positive and negative) screening results to families (including confirmatory testing) to inform clinical care?
Can findings be returned safely and effectively using the approach adopted for the study overall?
Are the specific clinical pathways established for the disorders included in the study being followed?
Can we sustainably collect data on important outcomes for babies and families, and is the process of collecting this outcome sustainable to families?
Impact
What is the impact of the programme on families, stakeholders and the wider system?
What is the impact on the wider family and society including: family member case identification, reproductive choices and patient and public values and preferences?
Test performance and clinical utility
What is the clinical utility of genomic newborn screening as judged by the number of apparent true positive screening diagnoses identified?
What proportion of apparent false positive and false negative findings are there in the study according to each condition’s working case definition?
What proportion of babies have uncertain status following orthogonal testing according to each condition’s working case definition?
What age are looked for conditions clinically diagnosed and treatment started with genomic newborn screening as compared to standard of care alone?
What are the demographic factors that determine outcome e.g., percentage apparent true positive findings?
REFERENCES
