Evaluation of Reliability, Repeatability, Robustness, and Confidence of GPT-3.5 and GPT-4 on a Radiology Board–style Examination
Satheesh Krishna, Nishaant Bhambra, Robert Bleakney, Rajesh Bhayana
Abstract
Default GPT-3.5 and GPT-4 were reliably accurate over three attempts at answering radiology board–style multiple-choice questions but had poor repeatability and robustness and were frequently overconfident, limiting usability without domain-specific optimization.
Topics & Concepts
MedicineRepeatabilityConfidence intervalReliability (semiconductor)Robustness (evolution)Medical physicsRadiologyStatisticsInternal medicineBiochemistryPower (physics)PhysicsQuantum mechanicsGeneChemistryMathematicsArtificial Intelligence in Healthcare and EducationRadiology practices and educationRadiation Dose and Imaging