Interobserver and intraobserver reliability for 2 grading systems for gastric ulcer syndrome in horses
Jessica C. Wise, Edwina Wilkes, Sharanne Raidal, Gang Xie, Danielle Crosby, Josephine Hale, Kris Hughes
Abstract
BACKGROUND: Grading of equine gastric ulcer syndrome (EGUS) is undertaken in clinical and research settings, but the reliability of EGUS grading systems is poorly understood. HYPOTHESIS/OBJECTIVES: Investigate interobserver and intraobserver reliability of an established ordinal grading system and a novel visual analog scale (VAS), and assess the influence of observer experience. ANIMALS: Sixty deidentified gastroscopy videos. METHODS: Six observers (3 specialists and 3 residents) graded videos using the EGUS Council (EGUC) system and VAS. Observers graded the videos three 3 for each system, using a cross-over design with at least 1 week between each phase. The order of videos was randomized for each phase. METHODS: Interobserver and intraobserver reliability were estimated using Gwet's agreement coefficient with ordinal weights applied (AC2) for the EGUC system and the intraclass correlation coefficient (ICC) for the VAS. RESULTS: Using the EGUC system, interobserver reliability was substantial for squamous (AC2 = 0.69; 95% confidence interval [CI], 0.57-0.80) and glandular mucosa (AC2 = 0.72; 95% CI, 0.70-0.75), and intraobserver reliability was substantial for squamous (AC2 = 0.80; 95% CI, 0.71-0.90) and glandular mucosa (AC2 = 0.80; 95% CI, 0.74-0.86). Interobserver reliability using the VAS was moderate for squamous (ICC = 0.64; 95% CI, 0.31-0.96) and poor for glandular mucosa (ICC = 0.35; 95% CI, 0.06-0.64), and intraobserver reliability was moderate for squamous (ICC = 0.74; 95% CI, 0.62-0.86) and glandular mucosa (ICC = 0.56; 95% CI, 0.39-0.72). CONCLUSIONS AND CLINICAL IMPORTANCE: The EGUC system had acceptable intraobserver and interobserver reliability and performed well regardless of observer experience. Familiarity and observer experience improved reliability of the VAS.