Evaluating the Quality and Safety of Ambient Digital Scribe Platforms Using Simulated Ambulatory Encounters

Abstract

Objective To evaluate and compare the quality and safety of ambient digital scribe (ADS) platforms using simulated ambulatory encounters. Methods Five ADS platforms were evaluated using audio recordings of fourteen simulated clinical encounters. Audio recordings were played on a laptop computer and captured by ADS platforms on a mobile phone. Generated transcripts were compared to professional transcriptions. Clinical notes were graded using rubrics of key elements for each case. Note errors were classified as omission, commission, or partially correct. Potential clinical harm was assessed using the agency for healthcare research and quality harm scale. Note quality was assessed using the 9-item Physician Documentation Quality Instrument (range 9-45). Statistical comparisons included Friedman and χ 2 tests with a correction for multiple comparisons. Results Transcripts generated by platforms A through D contained an average of 13.9 (95% CI, 6.0-17.5) errors, with 19.5% of the transcript errors transmitted to the clinical note (95% CI, 6.6%-28.8%). For clinical notes, mean percent error across platforms was 26.3% (95% CI, 17.0%-31.0%) with a significantly higher proportion of errors in notes generated by platform E ( P <.0053 for all comparisons). Of correctly reported elements, only 35.8%±11.3% were consistently correct across all platforms. An average of 3.0 (95% CI, 0-4, range 0-21) errors per case had potential for moderate-to-severe harm. The mean physician documentation quality instrument–9 score was 36±4, with significant variation between platforms. Conclusion Clinical notes generated by ADS platforms using simulated encounters reports important inter-platform and intra-platform variability in accuracy and quality. These findings indicate a need for standardized, objective evaluation and reporting.

Topics & Concepts