Lessons Learned in Building Expertly Annotated Multi-Institution Datasets and Hosting the RSNA AI Challenges
Felipe Kitamura, Luciano M. Prevedello, Errol Colak, Safwan S. Halabi, Matthew P. Lungren, Robyn L. Ball, Jayashree Kalpathy–Cramer, Charles E. Kahn, Tyler Richards, Jason F. Talbott, George Shih, Hui Ming Lin, Katherine P. Andriole, Maryam Vazirabad, Bradley J. Erickson, Adam E. Flanders, John Mongan
Abstract
The Radiological Society of North America (RSNA) has held artificial intelligence competitions to tackle real-world medical imaging problems at least annually since 2017. This article examines the challenges and processes involved in organizing these competitions, with a specific emphasis on the creation and curation of high-quality datasets. The collection of diverse and representative medical imaging data involves dealing with issues of patient privacy and data security. Furthermore, ensuring quality and consistency in data, which includes expert labeling and accounting for various patient and imaging characteristics, necessitates substantial planning and resources. Overcoming these obstacles requires meticulous project management and adherence to strict timelines. The article also highlights the potential of crowdsourced annotation to progress medical imaging research. Through the RSNA competitions, an effective global engagement has been realized, resulting in innovative solutions to complex medical imaging problems, thus potentially transforming health care by enhancing diagnostic accuracy and patient outcomes.