Nonresponse adjustment using clinical and perioperative patient characteristics is critical for understanding post-discharge opioid consumption

Survey-Based and EHR-Based Data Collection

Patients who underwent a surgical procedure at Beth Israel Deaconess Medical Center (BIDMC) from March 1, 2019 to October 31, 2020 were prospectively surveyed after discharge to quantify opioids consumed and to measure satisfaction with pain control. Patients were included if they were 18 years old or greater, spoke English, had a valid cell phone number, and underwent any surgery at BIDMC during the study period. Patients were excluded if they were: trauma patients, patients undergoing multiple surgeries during their index hospitalization, hospitalized for greater than 2 weeks after their surgery, discharged to a rehab facility, had already participated or were readmitted prior to completing the survey, incarcerated patients, or expired patients. An initial SMS message was sent 14 days after discharge, and up to 2 reminder messages at 7-day intervals. Specific survey content is included in Supplemental Table 1. The survey was hosted using Research Electronic Data Capture (REDcap) software.¹⁹ Patients’ post-discharge opioid consumption was measured in one of two ways: 1) they reported how many pills were remaining from their prescription in the survey, or 2) no opioid prescription was recorded in the EHR.

The primary outcome was the number of opioid pills consumed by the patient after discharge from the surgical procedure, converted to morphine milligram equivalents (MMEs). Initial opioid prescription size, patient demographics, comorbidities, hospitalization status, and surgical- and anesthesia-related characteristics were programmatically extracted from the EHR. The purpose of extracting these EHR-based variables was to comprehensively perform nonresponse bias adjustment of our survey results so that we could more accurately estimate MME consumption. A summary listing of the 40 predictors is provided in Supplemental Table 2.

To provide a reference for comparison for our SMS-to-web survey results, we used a prospective database of opioid consumption data from a prior telephone-based survey at our institution. For this survey, patients who underwent surgery at our institution from October 2017 to June 2018 were called within a month after their surgery and asked how many pills remained. Patients who were still taking opioids were called at 2-week intervals for up to 3 calls in order to record the final tally of pills consumed. Of 3,302 contacted patients, 1,980 (60%) fully answered phone surveys about their post-discharge opioid consumption. Additional details are provided in a previous study.⁸

Statistical Analysis

Variation in response by EHR characteristics was evaluated using chi-squared tests. Continuous variables were grouped into discrete bins based on common clinical thresholds (e.g. BMI) or based on sample quantiles (e.g. age) so that different subgroups of these variables could be displayed; the Kolmogorov-Smirnov (K-S) test results on raw continuous variables were comparable. Missing predictor data was imputed using generalized low-rank models^20,21, and missingness indicators were included as predictors after eliminating perfectly collinear indicators.²² Additional details on missing data imputation are included in the supplemental information.

Nonresponse bias is a primary threat to survey-based data collection, where estimates of the outcome distribution (opioid consumption in our case) are confounded by variables that are related both to probability of responding to the survey and to the outcome.^23,24 We conducted a targeted learning (TL)-based double-robust analysis that reduces confounding bias in three ways: 1) inverse probability weights based on a propensity score estimate, 2) an outcome regression adjustment that estimates the relationship between patient characteristics and opioid consumption, and 3) a targeting step that improves estimation using the equation of the specific statistical parameter being studied (i.e. quantiles in our case).^25,26 We used this TL approach because quantile estimation tasks like the ones we performed in this study (i.e., providing an estimation of the median and 75th percentile of opioid consumption) based on targeted learning has shown efficiency gains equivalent to an increased sample size of 100% or more.²⁷ In other words, TL adjustment may improve the accuracy of quantile estimations as much as doubling our sample size would.

Although adjustment can be performed with simpler logistic regression or linear probability models, their linear formulas oversimplify the complex relationship between predictor variables and our two outcomes of interest, opioid consumption and survey response.^28,29 Accepting this misspecification bias may distort consumption estimates further from the truth, leading to potential patient harm. Machine learning-based estimators have the potential to more accurately capture complex nonlinear relationships between predictor variables and survey response, thereby minimizing residual bias.³⁰ We used an ensemble machine learning approach, called the Super Learner algorithm, to evaluate multiple prediction algorithms for their accuracy in capturing these relationships, and to create a weighted average designed to achieve the best accuracy.^31–33 These modeled relationships were then used to generate quantile estimations using TL as described above. We evaluated the relative importance of predictors by tallying how frequently they were used in decision tree splits of the best performing machine learning algorithm, Bayesian additive regression trees (BART).³⁴ Additional estimation details are provided in the supplemental information.

Prior Opioid Consumption Surveys

Nonresponse bias is a challenge for any survey-based data collection.²³ Clinical studies often overlook the issue of nonresponse and do not adjust for it;^7,36 failing to do this prior to interpreting survey results can lead to biased results and non-generalizable conclusions. Few opioid consumption studies have examined any factors associated with nonresponse. A recent study measuring post-discharge opioid consumption among patients undergoing orthopedic and urologic surgery using an SMS-based survey found that the presence of comorbidities were associated with nonresponse, however nonresponse adjustment was not performed.¹⁸ To our knowledge, no opioid consumption surveys have adjusted their consumption estimates for nonresponse bias.^11,12,18 The accuracy and generalizability is largely unknown for surveys in which nonresponse bias has not been examined.

Importance of EHR-based Adjustment

We found that several EHR-based clinical and perioperative factors were associated with both opioid consumption and survey response in our study, suggesting that unadjusted estimates of opioid consumption may be biased. Multiple prior studies confirm this overlap: age, race, smoking status, and comorbidity burden are associated with MME consumption.^11,12,35 The association of these clinical and perioperative predictors with both response and consumption is confounding and may lead to inaccurate opioid consumption estimates if not incorporated into adjustment, and may adversely affect patient care if used to guide opioid prescribing recommendations. Nonresponse bias adjustment using these factors significantly revised the median and 75th percentile opioid consumption estimates for most procedures. Several prior clinical surveys on other topics have uncovered associations between EHR-based clinical factors and survey response as well.^13–17We also used EHR data to improve the completeness of our consumption data, instead of relying on survey-reported opioid consumption alone. Patients who did not respond to the survey but were not prescribed any opioids in the EHR were counted as consuming zero MMEs in our final dataset. In merging survey data with EHR-based patient outcome data we were able to increase the number of patients for whom opioid consumption could be measured.

Importance of Machine Learning-Based Adjustment

Traditional parametric methods for nonresponse bias adjustment such as logistic regression are limited in their ability to correct for this bias because they do not fully capture the complex patterns that differentiate responders and nonresponders.^28,29 The bias that is accepted by these simple models may distort opioid consumption estimates so that they are further from the truth, leading to harm by misinforming prescribing guidelines. Ensemble machine learning, as we used in this study, can reduce estimation bias compared to logistic regression by relying on algorithms that best predict the outcome and response variables.^30,37 In observational studies at risk of confounding bias, it is a best practice to use machine learning-based adjustment to generate the least biased estimates. Future surgical and opioid research should incorporate these recent methodological developments.³⁸

Limitations

While our prior telephone-based survey results served as the standard of comparison for our SMS-to-web survey results, it is important to note that there is no “gold standard” survey method for measuring opioid consumption; telephone-based consumption data is vulnerable to nonresponse bias as well. Furthermore, there may have been background trends that influenced opioid prescribing habits and opioid consumption at our institution between 2017-2018 when the telephone data was collected, and 2019-2020 when the SMS-to-web data was collected. As our study and prior work has identified, one of the strongest predictors of opioid consumption is prescription size.¹¹ The difference between SMS-to-web consumption estimates and telephone consumption estimates, even after nonresponse adjustment, may be partially explained by these two phenomena. Nonetheless, we chose to use our telephone-based consumption data as a reference for comparison since it is a commonly used method for collecting opioid consumption data in prior studies, had a high response rate of 60% in our study, and its accuracy has been validated by previous work demonstrating that it aligns closely with in-person based consumption estimates.^8,9,11,12 While our SMS-based survey had a comparatively lower response rate as expected,^18,39,40 the automated nature of this survey method may limit recall bias. We were able to automatically contact patients on a weekly basis after discharge, while studies using telephone and other labor-intensive survey methods have waited up to 12 months after discharge to survey patients on their pain and opioid consumption after surgery.⁷