AI meets sleep surgery: assessing drug-induced sleep endoscopy interpretation with a large language model

Abstract

Abstract Study Objectives To evaluate the performance and safety of a large language model in interpreting drug-induced sleep endoscopy (DISE) videos and providing treatment recommendations for obstructive sleep apnea, compared to expert human raters and the contemporaneous clinical report. Methods This prospective, blinded study included 16 adults undergoing drug-induced sleep endoscopy at a tertiary academic center. For each case, an anonymized procedural video, clinical vignette, and examination findings were independently reviewed by two sleep surgery experts, a senior otolaryngology resident, and a large language model. All the raters assessed the video quality, airway maneuvers, airway collapse at each anatomical site, and recommended therapy. Concordance with the clinical reference standard was evaluated using Cohen’s kappa and intraclass correlation coefficients. Safety and reproducibility were assessed through subgroup and error-type analyses. Results Human raters demonstrated perfect or near-perfect agreement with the reference standard for the velum, oropharynx, tongue base, and jaw thrust response, and moderate agreement for epiglottic collapse. The large language model matched expert performance for all domains except the epiglottis, where moderate agreement was observed. Model-generated treatment recommendations were safe, consistent with accepted clinical practice, and highly reproducible between independent runs. No unsafe or discordant recommendations were identified. Conclusions In this single-center study of 16 patients, a large language model accurately interpreted DISE videos and generated safe recommendations consistent with accepted clinical practice, approximating expert performance in most domains. Larger multicenter cohorts are needed to validate these findings and confirm generalizability. Statement of Significance This study demonstrates that artificial intelligence can interpret complex airway videos in sleep surgery and provide safe, expert-level treatment advice. By comparing the performance of a large language model with that of experienced clinicians, our findings suggest that advanced technology can help standardize decision-making in a highly subjective area of care. This work highlights the promise of artificial intelligence as an adjunct to clinical judgment, especially in settings where expert access is limited. However, important questions remain about the use of artificial intelligence in challenging cases and its integration into real-world practice. Future research should focus on larger, more diverse groups of patients and on ensuring ongoing oversight and safety.

Topics & Concepts