Precision Electroconvulsive Therapy (PET) project: tackling (cost) effectiveness and patient’s perspectives

WP1: Evaluation of effectiveness and safety of ECT and its predictors

This work package will realize two aims: (1) investigate the effectiveness of ECT and determine its clinical predictors in the Netherlands, (2) investigate cognitive side-effects of ECT and determine its clinical predictors in the Netherlands, (3) determine the optimal position of ECT in the depression treatment algorithm.

Participants and study design of WP1

A database —the Dutch ECT Cohort (DEC)— will be compiled from available clinical and research cohorts of patients that received ECT for (moderate) severe depressive episodes in the Netherlands during the last 20 years. ECT data will be retrieved from all three sorts of health organizations providing ECT in the Netherlands (i.e., general hospitals [GH], university medical centers [UMC] and institutions for mental health care [MHI]) to gather data representative for the current Dutch ECT practices. Twelve large ECT-centers will provide their coded databases regarding the essential variables for this study. A total of approximately 1500 patients will be included in the database. As has also been approved by the ethical board, patient burden due to this study is absent because archival data will be collected and patients have consented for the original studies. For the patients in the clinical databases, an opt-out was offered in cases where the correct contact information was available upon advice of privacy officers. If active opt-out was not feasible e.g. because of lacking contact information or database size, local privacy officers performed a data protection impact assessment according to GDPR guidelines.

Data-extraction protocol

About half of the data (n=684) will come from local research cohorts for which complete variables of interest are available (Birkenhager et al, 2019; Dols et al., 2017; Guloksuz et al., 2015; Heijnen et al., 2019; Oudega et al., 2010; Scheepens et al., 2020; van Eijndhoven et al., 2016; van Oostrom et al., 2018; van Waarde et al., 2015; Verwijk et al., 2013). Next, additional data will be derived from clinical cohorts of our national partners as cited above. At these involved sites, patient data have either been stored in prospectively collected clinical databases or in their respective electronic patient files (EPF) which will allow digital data collection for several key parameters (n=1104).

All sites evaluate severity of depression by either systematically applying the Hamilton Depression Rating Scale (HDRS; Hamilton 1967), the Montgomery-Åsberg Depression Rating Scale (MADRS; Montgomery & Åsberg, 1979) or the Inventory for depressive Symptomatology (IDS; Rush et al, 1986). Generally, these scales are administered on a weekly basis during ECT-courses. These scales apply validated cut-off values to determine response, remission or non-response of depression, thus enabling a combined analysis for the respective outcomes. Subitems scores of these scales will be collected as well. Furthermore, described treatment outcome in the discharge letter was also assessed. Table 1 gives an overview of the clinical characteristics that will be included in our database based on previous meta-analyses (Haq et al., 2015; van Diermen et al., 2018, Peeters et al., 2016).

For each subject, only the first course of ECT in the electronic database will be included, preventing enrichment for individuals repeatedly treated with ECT as a result of previous good response to ECT. Several variables specific for ECT will be included for descriptive purposes such as applied electrode placement at the end of the ECT-course (right-unilateral [RUL], bifrontotemporal [BT] of bifrontal [BF]), total number of ECT-sessions applied during the ECT-course, assessment of seizure adequacy during the course (operationalized by the presence of a mean duration of motor (EMG) seizures >= 15 seconds or electroencephalographic (EEG) seizures >= 25 seconds), used anesthetics, and drop-out during the ECT-course due to clinical reasons or side-effects (e.g., headache, muscle aches, postictal delirium, nausea).

Analyses

Our primary outcome is remission, operationalized as a depression score on the HDRS-17 of 7 or less or on the MADRS of 9 or less at the end of the ECT-course. Our secondary outcomes are response and the presence of ‘clinically significant cognitive side-effects’, operationalized as a broad composite score of the different available test instruments used. Patients’ cognitive performance will be quantified using reliable change index (RCI) methods, to calculate the presence or absence of clinically significant decline in cognitive performance after ECT. A score of RCI = -1.645 is seen as a clinically significant decline (Jacobson & Truax, 1991).

Aim 1. Investigate the effectiveness of ECT and determine its clinical predictors

We will investigate possible well-known predictors (i.e., psychotic features, an older age, shorter duration of the index episode, absence of medication resistance, age at onset, psychiatric or somatic comorbidities) for remission (primary outcome) and cognitive side-effects (secondary outcome) controlling for confounders (sex) and collinearity. We will also control for referral bias as for example psychotic features are associated with high remission rates and prompt referral for ECT. Primary outcome measures will be on pre-post effects of ECT on depression.

Given the complex nature of the merged data and in line with earlier work from our group members, machine learning approaches will also be used to analyze the data. To obtain the most optimal model, the following approach will be used to analyze the data:

• - A priori identification of possible predictors: existing literature and expert opinion will be used to see which predictors likely play an important role in achieving remission and whether additional feature engineering (i.e., creating more informative variables from existing variables) is required.

• - Splitting the data in a training- and test-set: The estimated size of the dataset allows us to split into a training-set and a test-(hold-out) set to assess model accuracy. A training-set is a random subset of the data that was used to train the various models. A testing-set is a random subset of the data (mutually exclusive from the training-set) that is used to assess performance of the models on new data.

• - Fit various statistical learning models on training-set: Next, different statistical learning techniques will be investigated for their applicability on the data. Common techniques are (linear/logistic) regression, support vector machine models, random forests or neural networks. The different models will be evaluated on their ability to predict remission in patients by cross-validating the models. Cross-validation is a technique to estimate the accuracy of a model by repeatedly partitioning the data and train the data on the training-subset and validate on the other.

• - Evaluating the model on the test-set: The model with the best cross-validated performance will be used to predict disability weights for every individual in the testing-set.

• - Determine plausibility and clinical relevance with patients and experts: After determining the final model, patient profiles and indicators for early ECT will be drafted in agreement with the various participating experts. These results will also be used in WP3.

Aim 2. Investigate cognitive side-effects of ECT and determine its clinical predictors

In a subset of the DEC (n=576), more extensive data on cognitive functioning before and after ECT (directly after the course or within the first year) will be available and will be used to 1) examine the evolution of cognitive performance over time following ECT and 2) to identify clinical and demographic predictors for the risk of developing cognitive side-effects following ECT. By calculating norm-scores of the various tests within one of the four main cognitive domains (attention, memory, executive function and language) results of various cohorts on numerous patients will be obtained in more detail with more power. Presence or absence of individual clinically significant cognitive decline or improvement will be operationalized by calculating the reliable change index (RCI, Jacobsen & Truax 1991). Pre- and post-ECT cognitive scores of neuropsychological data will be taken as a starting point for cognitive change.

Aim 3 Determine the optimal position of ECT in the depression treatment algorithm

For this aim, the effectiveness of ECT ‘earlier’ versus ‘later’ in the patient’s treatment journey will be determined while controlling for possible confounding factors.

The timing of ECT (‘earlier’ vs ‘later’) will be operationalized by categorizing patients by progressive completion of the five steps of the European and Dutch Guideline for Depression (i.e., first selective serotonin reuptake inhibitor [SSRI], second SSRI, tricyclic antidepressant [TCA], lithium addition and irreversible mono-amine oxidase [MAO] inhibitor, respectively) before receiving ECT. Using ECT at an early treatment stage will be defined as having received 0 to 2 adequate pharmacological treatment attempts, while ‘late’ usage will be defined as having undergone > 2 adequate pharmacological treatment courses.

The dichotomization of the DEC cohort in early ECT vs late ECT will allow us to compare the relative efficacy as a function of the level of treatment resistance and inverse propensity weighting will be used to compare these cohorts. Inverse propensity weighting (also known as inverse probability of treatment weighting) is a statistical technique that could substitute for randomization to treatment conditions when the latter is not feasible (Thoemmes & Ong, 2016; Austin, 2008). Based on baseline variables such as age, sex, severity of depression and presence of psychotic symptoms, weights will be estimated to help to make the cohorts comparable with respect to prognostically relevant patient characteristics, and only differing on characteristics related to the treatment resistance and duration of the index period.

Choosing the best set of covariates will be done in a data-driven (empirical) way by checking if baseline balance is achieved after weighing with regard to prognostically relevant variables. The propensity weights will be incorporated in the statistical analysis of outcome parameters by means of inverse propensity score weighted generalized mixed models. This will deliver intention-to-treat (ITT) estimates of efficacy. Logistic regression analysis, including ‘ECT in early treatment stage yes/no’ as predictor and selected variables associated with P-values under 0.10 in the univariate analyses (see Aim 1) will be performed with the primary and secondary outcomes as dependent variables. All tests will be two-sided, with P value set to 0.05 to infer statistical significance. Although these observational data will not permit a conclusion on causal inferences, this design will allow for a comparison of effect-sizes (ES) of early vs late ECT (e.g., early ECT shows a larger ES, both early and late ECT have equal ES, late ECT shows a larger ES).

WP2: Prospective assessment of (cost)-effectiveness and side-effects of depressive patients treated with either ECT or TCA

For a complete economic evaluation of ECT in the Netherlands, we set two aims: (1) determine cost-effectiveness of providing ECT compared to medication by determining the impact of ECT on remission rates, side-effects and quality of life using A) a one-year time horizon (trial-based economic evaluation) and B) a lifetime time horizon (model-based economic evaluation); and (2) estimate the impact on cost-effectiveness of providing ECT as an earlier versus a later treatment step on remission rates and quality of life.

To determine the cost-effectiveness of ECT compared to medication, a trial-based (prospective) and model-based health economic evaluation will be conducted in accordance with the Dutch guidelines for economic evaluations (Zorginstituut Nederland, 2016). Within both approaches the impact of providing earlier vs later ECT will be integrated. Health economic evaluations will be performed from both the health care system’s perspective as the societal perspective (Zorginstituut Nederland, 2016).

A. Trial-based economic evaluation in WP2

We will conduct an economic evaluation alongside a multisite, observational prospective cohort study (PROSPECT) with two parallel inception cohorts of patients with major depressive disorder who will be treated with either ECT or TCA. This prospective cohort design is chosen because a randomized controlled trial is not feasible and because it better reflects real-life conditions for comparative cost-effectiveness analyses. Moreover, it is more pragmatic to monitor sustained effectiveness, adverse events and cognitive side-effects during longer follow-up periods after start of ECT or antidepressants, which are important factors to consider for judgements about clinical benefits and cost-effectiveness. The risk of bias is an important limiting factor in observational studies. In this context, ECT may be initiated for more severe conditions (confounding by indication), the cohorts may differ in treatment preferences and adherence to treatment protocol (selection bias), and the cohorts may differ in relapse rates at different times after the initial treatment period (time-varying confounding). To cope with the risk of biases, PROSPECT is setup in accordance with the ROBINS-I tool (http://www.riskofbias.info), specifically designed for studies with a cohort-type of design, in which individuals who are receiving different interventions are followed up over time. The ROBINS-I tool assesses seven domains in which bias might occur in non-randomized intervention studies, and which can be addressed by careful consideration of the risks of bias, systematic assessment of confounding factors, a detailed treatment protocol and explicit reporting guidelines (including a clinical trial registry at clinicaltrials.gov identifier: NCT05306184; Sterne et al., 2016). To test for biases in the observational design, here we will also use inverse propensity weighting to compare the two prospective cohorts (Austin, 2008).

Data-collection

110 patients (>17 years) with a unipolar MDD who will start ECT and 110 patients (>17 years) with a unipolar MDD who will undergo treatment with antidepressants will be recruited from specialized clinics (i.e., GH, UMC and MHI) and followed-up for one year to gather reliable data on efficacy including relapse rates, cost-effectiveness and quality of life. For an overview of all collected data at the different timepoints see table 2.

The Structured Clinical Interview for DSM 5 disorders (American Psychiatric Association, 2017) will be used to assess diagnosis. The Dutch Method for quantification of Treatment Resistance in Depression (DM-TRD) (Peeters et al., 2016) will assess and quantify the level of treatment resistance. The Hamilton Depression Rating Scale 17 item version (HDRS-17), an observer rated scale to assess the severity of depressive symptoms (Hamilton, 1967), will be assessed to measure the severity of depression.

In keeping with the other work packages, criteria for inclusion in the prospective cohorts will be:

• - adult patients (>17 years) with a major depressive disorder who will either start with ECT or TCA;

• - failed response to at least one adequate dose-duration trial with antidepressants;

• - moderate or severe depression (HDRS-17 >16). Exclusion criteria will be:

• - lifetime diagnosis schizophrenia or schizoaffective disorder, current substance abuse disorder, organic brain syndrome;

• - the presence of a concurrent significant somatic condition impeding the ability to participate.

In- and exclusion criteria will be checked by psychiatrists from the participating centers. Patients will be enrolled in the groups via usual clinical care. If the patient is eligible, a researcher will provide information for this trial. Within a week, informed consent will be obtained and a personal overview of the study-set-up will be given. We will use CASTOR as a validated data management tool, available for all researchers of the collaborating institutes, and will appoint a monitor from the Clinical Research Centre Nijmegen, Radboudumc, who is certified in Good Clinical Practice and fulfills the requirements for monitoring clinical trials. Inclusion rate will be monitored through https://zorgevaluatienederland.nl/evaluations/prospect

Intervention(s)

ECT will be applied following a treatment protocol in accordance with the Dutch guideline on ECT for depression (Van den Broek et al., 2021). Treatment with TCA will consist of the protocolled pharmacological treatment step, in accordance with the medication algorithm of the Dutch multidisciplinary guideline for depression (Spijker J, 2013).

Primary outcome

Depression severity: The Hamilton Depression Rating Scale 17 item version (HDRS-17; Hamilton 1967) after eight weeks of initial treatment and at regular three months intervals in the first year.

Secondary outcomes

Quality of life: the EuroQol-5D (EQ-5D-5L; Herdman et al., 2011) will be assessed with a validated health-related quality of life (HRQoL) instrument directly after eight weeks of treatment, after three, six, and twelve months. The EQ-5D-5L investigates health states in five different domains: i.e. mobility, self-care, usual activities, pain/discomfort and anxiety/ depression. Current Dutch tariffs will be applied to each of the health states to calculate utilities, anchored at 0 (death) and 1 (full health). Resulting EQ-5D utilities will be merged over time using the trapezium rule resulting in QALY health gains over the entire twelve-month period.

Costs: The Trimbos Institute and iMTA Cost questionnaire for Psychiatry (TiC-P, Bouwmans et al., 2013) will be used to investigate the health care usage and associated costs. The TiC-P is most frequently used for healthcare receipt questionnaire for health-economic evaluation in the Netherlands. Four types of costs will be included: (1) intervention costs (i.e., ECT and TCA), (2) costs stemming from health care utilization, (3) costs for travel and informal care costs of patients and relatives, (4) costs due to loss of productivity (i.e. absenteeism and lesser efficiency in work), in paid work as well as volunteer jobs.

As central clinical endpoint for the cost-effectiveness analysis (CEA), the incremental costs per responder will be used (defined as 50% decrease on the HDRS-17). For the cost-utility analysis (CUA), change in quality of life will be calculated based on the individual pre- and post-treatment quality of life at the initial treatment period and at regular 3 months intervals in the first year.

Cognitive battery and assessment of side-effects

To measure cognitive functioning, several validated cognitive test instruments will be used to quantify the following cognitive domains: immediate and delayed verbal recall (anterograde memory), autobiographical memory and executive functioning. The tests will be used at baseline, directly after ECT, and at six and twelve months follow-up. Wherever possible, parallel versions are used to prevent learning effects. Tests that will be used are: Dutch Adult Reading Test (DART – premorbid IQ; only at baseline, Schmand et al. 1991); Montreal Cognitive Assessment (MOCA – general cognitive status; all time-points, Nasreddine et al. 2005); Category and letter fluency (language and executive functioning; baseline, 8 weeks and 6 months; Luteijn & Van der Ploeg, 1983; Mulder, Dekker, & Dekker, 2006); 15 woorden-test (verbal learning and memory; baseline, 8 weeks and 6 months; Saan & Deelman, 1986); Subjective Assessment of Memory Impairment (SAMI – subjective memory; baseline, 8 weeks and 6 months; Kumar, Han, Tiller, Loo, & Martin; 2016); Columbia Autobiographical Memory Interview (CUAMI – autobiographical memory; baseline, 8 weeks and 6 months; McElhiney, Moody, Sackeim, Hoogendoorn, & Bergfeld 1997; Verwijk, De Haan, Nwatarali, Hoogendoorn, & Bergfeld, 2021). This test battery takes about 60 minutes at baseline and 45 minutes at eight weeks and six months. The antidepressant side-effects checklist (ASEC; Uher et al., 2009) will be assessed at all timepoints to monitor the side-effects of the TCAs (Hodgson et al., 2015).

Course of depression

The life chart interview method will be used for detailed information on the course of depression (presence and severity of depressive symptoms) during the year after the start of treatment (after three, six and twelve months) (Lyketsos, Nestadt et al. 1994).

Sample size calculation

The statistical test will be conducted using a baseline adjusted ANCOVA with four follow-ups (or in an equivalently specified linear mixed model), because this is power-efficient. Taking account of an estimated effect size of 0.3, a correlation between measures of 0.30, we would require N=180 for testing at a=0.05 (2-sided) and a power of (1-ß)=0.80. We will increase sample size to compensate for loss to follow-up (20%) to N=220. The estimated effect size of 0.3 is considered moderate and conservative given that a recent meta-analysis performed by our group, as part of the update of the Dutch guidelines for ECT (van den Broek et al., 2021, demonstrated a standardized mean difference of 0.8 (0.29-1.29) between ECT and antidepressants based on HDRS-scores.

Analyses

Clinical outcome will be analyzed and reported in accordance with the CONSORT guidelines on an intention-to-treat basis. The sample consisting of all patients will be analyzed using linear mixed modeling. Linear mixed modeling will test intra-group and intergroup differences in effectiveness and side-effects. Level of treatment resistance within the index period will be used as a variable to investigate the effect of early vs late treatment with ECT. Relapse rates during follow up will be analyzed using logistic regression and compared between the two groups. Kaplan-Meier survival estimates will be computed on the basis of the outcome at eight weeks, three, six, nine and twelve months follow-up and the obtained scores will be compared using the log-rank test to account for response rate and time needed to reach 50% reduction of symptoms.

CEA and CUA: Cost calculations

The total costs will be computed from a bottom-up or micro-costing perspective, with units of health service multiplied by their appropriate unit cost price and added up to deliver an overall total cost estimate (Drummond, Sculpher, Torrance, O’Brien, & Stoddart, 2002). We shall make use of the standard unit cost prices as reported in the latest Dutch guideline for health economic evaluation (Zorginstituut Nederland, 2016, or later editions). Medication costs will be computed with prices based on the Daily Defined Dosage (DDD) taken from www.farmacotherapeutischkompas.nl and www.medicijnkosten.nl, while accounting for the pharmacist’s claw back. Loss of productivity will be assessed with the friction cost method according to the Dutch guideline. Informal care costs will be calculated on the base of shadow prices for unpaid work and the costs of transport will be computed as the mean distance per destination multiplied by standard cost prices. All costs will be expressed in 2018 euros and if needed will be updated to 2018 on the basis of the consumer price index.

The comparability of the groups will be assessed for costs and outcomes, and if necessary, there will be controlled for baseline differences (Manca, Hawkins, & Sculpher, 2005; van Asselt et al., 2009). We will impute missing cost and outcome data with single imputation based on predictive mean matching, and nested in non-parametric bootstraps for intention-to-treat (ITT) analysis (Brand et al., 2019). All cumulative costs and health gains during follow-up will be calculated with “the area under the curve method” to obtain the remission rate and cumulative QALY health gains as accrued over the measurements up to last follow-up at 6 months. The incremental cost-effectiveness ratio (ICER) will deliver the incremental costs per remitter and the incremental costs for each gained QALY. We will perform 5,000 non-parametric bootstraps and plot the simulated ICERs on the ICER plane to account for uncertainty. ICER acceptability curves will be plotted with various willingness-to-pay (WTP) setting to support judgements about whether ECT offers good value for money in comparison to treatment with TCAs. The robustness of findings will be assessed with sensitivity analyses (one-way) directed at uncertainty in the main cost drivers and outcomes (e.g. under different imputation strategies). All research findings will analyzed and reported in accordance with the relevant CONSORT and CHEERS guidelines (Campbell, Piaggio, Elbourne, & Altman, 2012; Husereau et al., 2013; Schulz, Altman, Moher, & the, 2010).

Exploratively, we will examine the impact on cost-effectiveness of providing ECT as an earlier versus a later treatment step on remission rates and quality of life. To explore the impact of early vs. late ECT, an incremental net monetary benefit regression (iNMBR) will be performed. To this extent, a regression model will be constructed to examine associations of treatment-specific characteristics (e.g., early or late ECT) with change net-monetary benefit, while controlling for selected patient-related confounders. Net monetary benefit (NMB) indexes the benefits per patients minus the costs per patient to convert the gain in QALYs into monetary values that express the net benefit.

B. Model-based health economic evaluation

The model-based health economic evaluation will be conducted according to recommended modelling guidelines (Briggs, Sculpher, & Claxton, 2006; Caro JJ, 2012) and will be informed by both the collected data from the prospective observational multi-site cohort study (PROSPECT) and the DEC cohort, that was described in WP1. The prospective study will inform important parameters not included in DEC such as resource use, relapse and quality of life at different timepoints in the first year. The purpose of the model is twofold: 1) to extend the time horizon of the trial-based health economic evaluation; and 2) to examine usual care in comparison to early or late ECT, given that it is currently considered an ethical challenge to randomize patients to early ECT.

The health-economic simulation will be performed in accordance with modeling techniques proposed by Briggs et al (2006) and follow the recommendations from ISPOR-SMDM Modeling Good Research Practices Task Force-2 (Roberts et al., 2012). The model compares costs between three scenarios: a base-case scenario representing usual / guideline congruent intervention for refractory depression and alternative scenarios representing the early or late intervention with ECT. Cost data will be extracted from the trial.

The long-term costs will be discounted in accordance with Dutch guidelines. The health-economic simulation model will conduct extensive uncertainty analyses overall cost and discounting parameters simultaneously and these will be reported accordingly. Using the model, a budget impact analysis (BIA) will be conducted in accordance with the ISPOR Task Group i.e. Mauskopf et al (2007) and Sullivan et al (2014) to investigate the impact health care budgets in case the new intervention is offered at different implementation levels (Mauskopf et al., 2007; Sullivan et al., 2014). The BIA will be conducted from various perspectives: (1) the perspective of the public purse and (2) the perspective of the health care insurer. For these perspectives the following scenarios will be investigated: a scenario in which ECT is offered to 40%, 60% and 80% of the eligible patients and an extreme scenario in which 100% of these patients will be receiving ECT. This will be done for early- and late-ECT separately to determine the potential increase in healthcare costs when offering more patients ECT, and what the implications are when ECT is offered earlier in the treatment trajectory. The different scenarios will be compared to a base-case scenario where ECT is offered in accordance with the current clinical guidelines for depressive disorder.

WP3: Qualitative analyses on ECT attitudes and experiences

The first aim of WP3 is to add qualitative analyses to the previously conducted quantitative data analyses about the treatment effects- and side-effects of ECT by conducting qualitative interviews with several stakeholders:

• - Patients and their relatives who have received ECT (including positive and negative outcome);

• - Patients and their relatives who were referred for ECT and chose not to receive ECT;

Study design: Explorative semi-structured interviews in 20 up to 24 patients and their relatives that were treated with ECT or did not choose ECT will be conducted. The ultimate goal is to clarify needs and wishes regarding information on ECT that facilitate shared-decision making. Interviews will also include experienced side-effects in patients that underwent ECT and anticipated side-effects in patients that did not choose for ECT.

Strengths and limitations

The project at hand may be seen as a national endeavor of a representative European country and we expect that our conclusions may be widely used. It addresses many open questions regarding ECT and will thus ultimately inform on its clinical use. There is a strong commitment of all partners to conduct this project. The collection of archival data is challenging, but this will ensure a large ‘real-world’ dataset on ECT practice with minimal burden on patients. However, we also have to note some limitations of this project. Firstly, we will not be able to randomize between different groups in the prospective part of this project. Moreover, follow-up of the prospective trial is only one year. On the other hand this was chosen as relevant follow-up duration in close collaboration with the patient union for depression and standard cost-effectiveness measures can be based upon this follow-up period.