Evaluating the Performance of Artificial Intelligence for Improving Readability of Online English- and Spanish-Language Orthopaedic Patient Educational Material
Carrie N. Reaver, Daniel E. Pereira, Elisa V. Carrillo, Carolena Rojas Marcos, Charles A. Goldfarb
Abstract
BACKGROUND: The readability of most online patient educational materials (OPEMs) in orthopaedic surgery is above the American Medical Association/National Institutes of Health recommended reading level of sixth grade for both English- and Spanish-language content. The current project evaluates ChatGPT's performance across English- and Spanish-language orthopaedic OPEMs when prompted to rewrite the material at a sixth-grade reading level. METHODS: We performed a cross-sectional study evaluating the readability of 57 English- and 56 Spanish-language publicly available OPEMs found by querying online in both English and Spanish for 6 common orthopaedic procedures. Five distinct, validated readability tests were used to score the OPEMs before and after ChatGPT 4.0 was prompted to rewrite the OPEMs at a sixth-grade reading level. We compared the averages of each readability test, the cumulative average reading grade level, average total word count, average number of complex words (defined as ≥3 syllables), and average number of long sentences (defined as >22 words) between original content and ChatGPT-rewritten content for both languages using paired t tests. RESULTS: The cumulative average reading grade level of original English- and Spanish-language OPEMs was 9.6 ± 2.6 and 9.5 ± 1.5, respectively. ChatGPT significantly lowered the reading grade level (improved comprehension) to 7.7 ± 1.9 (95% CI of difference, 1.68 to 2.15; p < 0.05) for English-language content and 8.3 ± 1.3 (95% CI, 1.17 to 1.45; p < 0.05) for Spanish-language content. English-language OPEMs saw a reduction of 2.0 ± 1.8 grade levels, whereas Spanish-language OPEMs saw a reduction of 1.5 ± 1.2 grade levels. Word count, use of complex words, and long sentences were also reduced significantly in both languages while still maintaining high accuracy and similarity compared with original content. CONCLUSIONS: Our study supports the potential of artificial intelligence as a low-cost, accessible tool to assist health professionals in improving the readability of orthopaedic OPEMs in both English and Spanish.