Discordance, accuracy and reproducibility study of pathologists’ diagnosis of melanoma and melanocytic tumors
Sarah Haggenmüller, Christoph Wies, Julia Abels, Jana Theres Winterstein, Lukas Heinlein, Carina Nogueira Garcia, Jochen Utikal, Sebastian A. Wohlfeil, Friedegund Meier, Sarah Hobelsberger, Frank Friedrich Gellrich, Mildred Sergon, Axel Hauschild, Lars E. French, Lucie Heinzerling, Justin Gabriel Schlager, Kamran Ghoreschi, Max Schlaak, Franz J. Hilke, Gabriela Poch, Sören Korsing, Cosimo Sarfert, Carola Berking, Markus V. Heppt, Michael Erdmann, Sebastian Haferkamp, Konstantin Drexler, Dirk Schadendorf, Wiebke Sondermann, Matthias Goebeler, Bastian Schilling, Jakob Nikolas Kather, Stefan Fröhling, Mar Llamas‐Velasco, Luis Requena, Gerardo Ferrara, María Teresa Fernández‐Figueras, Sylvie Fraitag, F. Max Müller, Hans Starz, Heinz Kutzner, Raymond L. Barnhill, Richard Carr, Kenneth S. Resnik, Stephan A. Braun, Tim Holland‐Letz, Titus J. Brinker
Abstract
Accurate melanoma diagnosis is crucial for patient outcomes and reliability of AI diagnostic tools. We assess interrater variability among eight expert pathologists reviewing histopathological images and clinical metadata of 792 melanoma-suspicious lesions prospectively collected at eight German hospitals. Moreover, we provide access to the largest panel-validated dataset featuring dermoscopic and histopathological images with metadata. Complete agreement is achieved in 53.5% of cases (424/792), and a majority vote ( ≥ five pathologists) in 90.9% (720/792). Considerable discordance is observed for non-invasive melanomas (complete agreement in only 10/73 cases). The expert panel disagrees with the local pathologists' and dermatologists' diagnoses in 14.9% and 33.5% of cases, respectively. This variability highlights the diagnostic challenges of early-stage melanomas and the need to reconsider how ground truth is established in routine care and AI research. Including at least two pathologists or virtual panels may contribute to more consistent diagnostic results.