Statistical Analysis Plan for the Stepped-wedge Cluster Randomized Controlled Trial of Electronic Early Notification of Sepsis in Hospitalized Ward Patients (SCREEN) : a study protocol for a stepped-wedge cluster randomized controlled trial

Study Design

The study is conducted in the 5 Ministry of National Guard Health Affairs (MNGHA) hospitals which share the same Electronic Medical Record (EMR) system (BESTCare, South Korea). This study is designed as a stepped-wedge cluster randomized controlled trial, which allows to sequentially deliver the study intervention to all trial clusters over a number of periods. We present a glossary of terms in line with the CONSORT extension for stepped-wedge cluster randomized trials(11) in Table 1. The cluster refers to the unit of randomization, which is the hospital ward, and we will refer to it in the rest of the document as “ward”. A list of ward-level and patient-level eligibility criteria is outlined in the study protocol.(5) Wards are randomized into 9 sequences of 5 wards each to receive the intervention. After a baseline period of 2 months, the intervention is implemented in a new sequence of five new randomly selected wards, until it is eventually implemented in all 9 sequences (45 wards) (Figure 1). A computer-generated non-stratified randomly allocated concealed list determines the order in which the wards receive the intervention. The randomization list was maintained with a research coordinator who was not involved in this trial, and the ward allocation remained concealed from the research and clinical teams throughout the study and was revealed for a given sequence only 1 month before the implementation of the intervention to allow training.

• Download figure
• Open in new tab

Intervention and Control Groups

The intervention included the implementation of an electronic alert system and the associated training, feedback and audit. The alert was developed in the electronic medical record (EMR), with the feature of being active (visible to treating team) or masked (inactive in EMR frontend for the treating team but active in the backend of the EMR). Once a ward is randomized to the intervention group, the alert system was activated. The intervention group constitutes of patients admitted to the wards with the active alert, and the control group constitutes of patients admitted to the wards with masked alert.

In the intervention group, once a patient meets the qSOFA criteria, an alert appears in EMR as a pop-message and appears also on a hand-held device carried by the charge nurse of the corresponding ward. The alert prompts the nurse to notify the physician and prompts the physician to assess the patient for possible sepsis. At the beginning of the study, we launched a hospital-wide sepsis awareness campaign in all 5 participating hospitals focusing on the importance of timely interventions for sepsis. We also provided in-service training sessions to the related medical and nursing departments at the beginning of the study and before implementation in each ward. We conducted regular webinars with the leaders of active wards. We also created an intranet page with educational resources (videos, presentations, documents, posters and related links) that explain the project and provides clinical guidance and resources. We developed a dashboard to display data for each active ward on the number of alerts and the percentage and time to acknowledgment by nurses and physicians. We have set 15 minutes as an over-stretch target for nursing acknowledgment and 30 minutes for physician acknowledgment and we have tracked these indicators on the electronic dashboards. The data on dashboards have been made available in real-time for nursing and medical managers of each ward. In addition, reports have been generated every two weeks of the same data and shared with the teams in the active wards. The hospital administration and quality management department has been engaged in the process of feedback and of improving the response time to alerts. The bedside management and assessment have been at the discretion of the treating team, and the project has adopted the 2016 Surviving Sepsis Campaign guidelines and the hour-1 bundle, but without specific monitoring of bundle compliance.(1, 2)

Study Population

The study has been conducted in the 5 Ministry of National Guard Health Affairs (MNGHA) hospitals: King Abdulaziz Medical City-Riyadh, King Abdulaziz Medical Center-Jeddah, and Prince Mohammed Bin Abdul Aziz Hospital – Al Madinah, King Abdulaziz Hospital - Al Ahsa, and Imam Abdulrahman Al Faisal Hospital – Dammam. A CONSORT diagram will be generated according to the CONSORT extension for stepped-wedge cluster randomized trials (CRTs).(11); for each sequence at each period, the number of clusters receiving the intervention, the average cluster size, and its variance, and the number of clusters not receiving intervention will be presented (Figure 1).

Sample Size

The sample size for this stepped-wedge cluster-randomized design was calculated for 45 clusters with 10 periods (including one baseline period) using Power Analysis and Sample Size (PASS) software (PASS 15 Power Analysis and Sample Size Software (2017). NCSS, LLC. Kaysville, Utah, USA, ncss.com/software/pass). After a baseline period, 5 new clusters (12) switch from the control group to the intervention group at the beginning of each subsequent period. Using historical data obtained from the development domain of the EMR for ward patients admitted from 01 July 2018 to 30 June 2019, we calculated a baseline in-hospital mortality rate by day 90 of 3.13%. Based on the same dataset, 18.3% of eligible ward patients had an alert based on qSOFA criteria(6) with in-hospital mortality of 8.16% compared to 2% in the non-patients. For sample size calculations, we made the following assumptions: A) the impact of the intervention on mortality occurs only in patients who have the alert, B) Only half of the patients with the alert have sepsis, C) 90% of deaths among the patients with the alert occurred among septic patients, D) early intervention resulting from the alert will reduce the in-hospital mortality by 50%, i.e. from 8.16% to 4.08% in patients with sepsis and would lead to an overall change in in-hospital mortality for the whole cohort from 3.13% to 2.46% yielding a relative risk of 0.79 and a risk difference of 0.67% which is the target effect size, D) 80% power using two-sided Wald Z-Test and significance level of 5% and, E) an intra-cluster correlation (ICC, a measure of the relatedness of cluster) of 0.22 as estimated from the same retrospective electronic database. As the primary analysis would be adjusted for random effect to account for the correlation between patients within the same cluster, we used the estimation variance (σ²), which is calculated from responses (P1, P2), as the within-cluster variance (σw²) as suggested by Hussey and Hughes (2007) and Hemming and Girling (2014).(13, 14) As such, a reduction of in-mortality in hospital by 90 days from 3.13% to 2.46% (relative risk of 0.79 and a risk difference of 0.67%) requires a total sample size of 65250 subjects (average of 1450 subjects per cluster with an average of 145 subjects per cluster per period). With all five hospitals combined, this is expected to require 20 months (2 months per period). There is no planned interim analysis.

Study cohorts

Reporting Alert Information

We will report the number of patients with at least one alert, and the number of alerts per patient, the time to first alert and the alert information criteria which led to the alert trigger (Table S2). We will report vital signs in the 12 hours pre-alert (systolic blood pressure, diastolic blood pressure, heart rate, temperature, and respiratory rate) as well as laboratory parameters in the 12 hours pre-alert (lactate, white blood cells, bilirubin, creatinine) and whether a culture of blood, respiratory, urine or other body fluids (pleural, ascitic, CSF, joint) were obtained and whether treatments (intravenous fluids and antibiotics) were received in the 12 hours pre-alert.

Reporting Process Measures and Post-alert Physiologic Parameters

We will compare the following process measures between the two groups: A) percentage of patients with lactate reported within 12 hours of alert if not reported in the 12 hours before alert and highest lactate value reported in the 12 hours after the alert, B) the percentage of patients with blood culture ordered in 12 hours if not performed in the 12 hours before the alert, C) percentage of patients with respiratory, urine and body fluid cultures ordered in 12 hours if not performed in the 12 hours before the alert, D) intravenous fluid administered in 12 hours after alert (yes, no), E) percentage of patients who were not on antibiotics in the 12 hours before alert and had new antibiotic administered within 3 and 12 hours of the alert, F) post-alert systolic and diastolic blood pressure: (lowest value in the 12 hours after the alert) and heart and respiratory rate (highest value in the 12 hours after the alert) (Table S2).

Study Outcomes

The primary outcome is defined as all-cause in-hospital mortality within 90 days. Other outcomes include hospital length of stay (LOS) (ITT cohort and alert cohort), transfer to ICU within 90 days (ITT cohort and alert cohort) and 14 days of alert (alert cohort), ICU-free days in the first 90 days (ITT cohort and alert cohort), critical care response team (CCRT) activation within 90 days (ITT cohort and alert cohort) and 14 days of alert (alert cohort), cardiac arrest within 90 days (ITT cohort and alert cohort) and 14 days of alert (alert cohort), the need for mechanical ventilation, vasopressor therapy and incident renal replacement therapy within 90 days (all in ITT cohort and alert cohort) and 14 days of alert (alert cohort). Balancing/ safety outcomes measures include antibiotic-free days, acquisition of multidrug-resistant organisms within 90 days in both groups (ITT cohort and alert cohort) and acquisition of Clostridium difficile infection within 90 days (ITT cohort and alert cohort) (Table S3 and S4).

Analysis of Primary Outcome

The primary outcome of all-cause in-hospital mortality by day 90 will be compared between the intervention group and control group at the individual level with the use of mixed-effect logistic regression model with jack-knife method to estimates standard errors to account for grouping within clusters.(13, 17) We will use a generalized linear mixed model with a binary distribution and a log-link function to estimate the relative risk as a measure of effect. We will include two levels of random effects to account for nested clustering within wards and periods and two levels of fixed effects; hospitals and COVID-19 ward status. The model will be selected as the best model with a unique covariance structure that produces the lowest Bayesian Information Criterion (BIC) value. The covariance structures that will be considered in the model are: first order of autocorrelation covariance structure, unstructured covariance structure, Toeplitz covariance structure and variance component structure (VC). We will use Satterthwaite method to adjust for denominator degree of freedom for tests of the fixed effects. The random coefficients will be modeled using G-side random effects and obtain the subject-specific estimates by defining the appropriate variance-covariance structure. The overdispersion of parameters will also be assessed by calculating the ratio of the Pearson chi-square statistic and its degrees of freedom. If this ratio exceeds 1, it indicates that the variability in data has not been properly modeled and that there is residual overdispersion due to misspecification of the conditional distribution. In case of no event reported in a cluster for a particular period, a firth correction will be used in the generalized linear mixed model to avoid the issue of quasi-separation. The Newton-Raphson optimization technique with ridging option might be used to help with the convergence of the model. Results will be expressed as relative risk with 95% confidence interval (CI). The primary outcome will be also analyzed similarly in the alert cohort. In case any models fail to converge, we will report ORs from mixed logistic regression. We will also estimate risk difference.(18)

Analysis of secondary outcomes

Categorical outcomes including ICU admission, CCRT activation, cardiac arrest, the need for mechanical ventilation, vasopressor therapy, incident renal replacement therapy, acquisition of multidrug-resistant organisms and Clostridium difficile infection will be compared between the intervention and the control group, in a similar model to the one used in the analysis primary outcome. We will include two levels of random effects to account for nested clustering within wards and periods and two levels of fixed effects; hospitals and COVID-19 ward status. Results will be expressed as relative risk with 95% CI.

Continuous outcomes including hospital length of stay (LOS), ICU-free days, and antibiotic-free days will be compared using a mixed effect Poisson model (Table S3). We will include two levels of random effects to account for nested clustering within wards and periods and two levels of fixed effects; hospitals and COVID-19 ward status. The model will be selected as best model with a unique covariance structure that produces the lowest Bayesian Information Criterion (BIC) value. The covariance structures that will be considered in the model are: first order of autocorrelation covariance structure, unstructured covariance structure, Toeplitz covariance structure and variance component structure (VC).” The random coefficients will be modeled using G-side random effects and obtain the subject-specific estimates by defining the appropriate variance-covariance structure. The results will be expressed as beta estimates with 95% CI.

Sensitivity and subgroup analyses

To address the concern about contamination, we will conduct a sensitivity analysis excluding all patients in the control group in the 90 days before crossing over to the intervention group. Because there are fewer than 50 clusters, we will conduct a sensitivity analysis using a small sample correction with the Kenward-Roger method.(13, 19, 20) We will conduct a sensitivity analysis adjusting for the following covariates: type of wards (medical, surgical, oncology and mixed), age, baseline systolic blood pressure, baseline respiratory rate, GCS and Charlson comorbidity index. For the latter analysis, we will use imputation for missing variables as outlined below. In addition, we will conduct also a complete case sensitivity analysis. We will conduct a sensitivity analysis excluding the periods in which wards were assigned as COVID-19 ward.

We will analyze the primary outcome of all-cause hospital mortality by day 90 across predefined subgroups using the same model of the primary analysis. The predefined subgroups include age=< 65 years and >65 years, patients with documented infection source (including ICD-10AU for pneumonia, urinary tract infection, skin and soft tissue infection, intra-abdominal infection or other infections) and patients with no documented infection or infection source, patients admitted to medical, surgical, oncology and mixed wards, alert within 48 hours of admission and after 48 hours of admission and patients admitted to COVID-19 and non-COVID-19 wards. (Table S6). Results of the test of interaction will be reported.

Handling dropouts and missing data

We do not expect to have missing observations in the variables required for the primary analysis. Given the nature of the study, missing observations are expected, for example, not all patients will have all laboratory tests and therapies within the narrow time windows defined in the protocol. Variables used in the descriptive analysis will not be imputed. In some of the secondary and sensitivity analyses, imputation will be used. Because Glasgow outcome score is not documented for patients with normal neurologic status, missing observations will be assigned a normal value for purposes of the model adjustment. Other variables used in the model adjustment model will be assessed and characterized in terms of their pattern (i.e., Missing Completely at Random, Missing at Random, Missing Not at Random). For Missing Completely at Random data, all analyses will be based on a list-wise deletion approach where observation will complete values will be only considered for analysis. For variables with values Missing at Random, multiple imputation techniques will be utilized to impute the missing values as suggested by Rubin.(21) And for variables with values Missing Not at Random, a pattern-mixture model technique will be used to impute the missing values.(22)

Graphical Presentation

The subgroup analyses will be displayed as a forest plot.

Adjustment for multiplicity

To adjust for multiple testing for secondary outcomes and subgroup analyses, we will use the False Discovery Rate (FDR) as described by Benjamini and Hochberg.(23) In this procedure all hypothesis tests will be sorted in descending order based on their calculated p-value. All hypothesis tests below an index K will be rejected where K is calculated as follows: where i =m, …, 1, m is the total number of tested hypotheses; q = 0.05. The multiplicity testing adjustment will also be done on confidence intervals by constructing 1-K*q/m CI for each selected parameter.(12)

Protocol deviations

We will report protocol deviations, if they occur, including failure to implement the intervention in a given ward, or wrong implementation of the intervention in a ward assigned to the control group. These deviations will be documented on the CONSORT flow diagram. The data from such wards will be analyzed according to the ITT principle.

Study governance

The study management and development committee is responsible for the overall management of the study, data management and maintenance of the trial master file and statistical master file following the Good Clinical Practice principles.