Identifying bias in models that detect vocal fold paralysis from audio recordings using explainable machine learning and clinician ratings

Introduction Detecting voice disorders from recordings of voice could allow for frequent, remote, and low-cost screening before costly clinical visits and a more invasive laryngoscopy examination. The goals of this study were to detect unilateral vocal fold paralysis (UVFP) from voice recordings using machine learning, to identify which acoustic variables were important for prediction to increase trust by relating such variables with known pathophysiology, and to determine model performance relative to clinician performance on the same recordings.

Methods Patients with confirmed UVFP through endoscopic examination (N=77) and controls with normal voices matched for age and sex (N=77) were included. Voice samples were elicited by reading the Rainbow Passage and sustaining phonation of the vowel “a”. The 88 extended Geneva Minimalistic Acoustic Parameter Set features were extracted as inputs for four machine learning models of differing complexity. SHAP was used to identify important features.

Results The highest median bootstrapped Area Under the Receiver Operating Characteristic Curve (ROC AUC) score was 0.87, which varied depending on model and task, and beat the top clinician performance (range: 0.74 – 0.81). When training machine learning models on clinician’s CAPE-V estimates, the maximum performance was 0.84. The most important variables included some related to UVFP pathophysiology such as mean MFCC1, mean F1 amplitude and frequency, and shimmer variability depending on model and task. Surprisingly, many UVFP recordings had higher intensity than controls, which was the most important variable for prediction. This was confirmed by having clinician’s rate the background noise as a proxy for an increase in microphone gain, which showed to be higher for UVFP than controls. This and UVFP patients compensating their soft voices with extra vocal effort in order to be heard are likely causes of a systematic bias that allowed the models to detect UVFP.

Conclusion Using the largest dataset studying UVFP to date, we achieve high performance from just a few seconds of voice recordings, surpassing expert clinicians’ performance. However, we uncovered bias which may occur in voice biomarker research any time individuals have a soft voice. Explainable machine learning and clinician ratings thus provide a mechanism to detect UVFP, uncover how acoustic variables characterize a specific pathophysiology, and reveal bias.