Diagnostic performance of artificial intelligence for histologic melanoma recognition compared to 18 international expert pathologists

Abstract

To the Editor: Currently, pathologic melanoma classification is based on the—inevitably somewhat subjective—integration of several histologic features.1Adamson A.S. Welch H.G. Machine learning and the cancer-diagnosis problem–no gold standard.N Engl J Med. 2019; 381: 2285-2287Google Scholar Thus, discordance between pathologists classifying the same lesions can be substantial, and objective assistance tools are needed. The classification of dermoscopic skin lesion images based on convolutional neural networks (CNNs) works well.2Tschandl P. Codella N. Akay B.N. et al.Comparison of the accuracy of human readers versus machine-learning algorithms for pigmented skin lesion classification: an open, web-based, international, diagnostic study.Lancet Oncol. 2019; 20: 938-947Google Scholar On a histologic level, our pilot studies provided a proof regarding the principle of CNN-based melanoma recognition using tiny sections of hematoxylin-eosin–stained digitized slides.3Hekler A. Utikal J.S. Enk A.H. et al.Pathologist-level classification of histopathological melanoma images with deep neural networks.Eur J Cancer. 2019; 115: 79-83Google Scholar,4Hekler A. Utikal J.S. Enk A.H. et al.Deep learning outperformed 11 pathologists in the classification of histopathological melanoma images.Eur J Cancer. 2019; 118: 91-96Google Scholar We compared the ability of CNNs with that of 18 international expert pathologists from eight different countries to discriminate melanomas and nevi in a less artificial setting using hematoxylin-eosin–stained whole-slide images. Ensembles of 3 individual CNNs were trained and tested using single hematoxylin-eosin–stained whole-slide images of 50 individual melanomas and 50 nevi labeled by a panel of 2 experienced dermatopathologists according to the standard practice to provide the “ground truth” (Supplementary Figs 1 and 2 available via Mendeley at https://data.mendeley.com/datasets/j87c9jshxy/1, Supplementary Table I available via Mendeley at https://data.mendeley.com/datasets/j87c9jshxy/1). The same 100 digitized slides were diagnosed using a web-based survey by 18 international dermatopathologists, each with at least 5 years of experience. With respect to the ground truth, the 18 individual pathologists achieved a mean sensitivity, specificity, and accuracy of 88.88% (SD = 6.66%), 91.77% (SD = 3.99%), and 90.33% (SD = 4.52%), respectively. Ensemble CNNs trained using slides with or without annotation of the tumor region as a region of interest performed at par with the experts (Fig 1) in terms of mean sensitivity, specificity, and accuracy (unannotated: 88% [SD = 0.0%], 88% [SD = 1.15%], and 88% [SD = 0.58%], respectively, and area under the curve [AUC] 0.95; annotated: 94% [SD = 0.0%], 90% [SD = 2.31%], and 92% [SD = 1.15%], respectively), with an AUC of 0.97. Majority of the votes of the expert panel yielded the best accuracy (98%). A statistical analysis of the performance differences is shown in Supplementary Table II (available via Mendeley at https://data.mendeley.com/datasets/j87c9jshxy/1). Only half (47%) of the diagnoses were unanimous. Overall, the discordance was 13.45%. Nine lesions were divergently classified by a third or more pathologists. Two were classified divergently based on majority (Fig 2, Supplementary Table I), of which 1 was originally classified as severely dysplastic acral nevus and the other as incompletely excised desmoplastic acral Spitz nevus with the melanocytic acral nevus with intraepidermal ascents of cells phenomenon, both of which the ensemble CNNs classified as melanoma. The diagnoses by ensemble CNNs trained using unannotated and annotated whole-slide images differed from the ground truth in 12 and 8, respectively, often pathologically unequivocal, cases. An ensemble CNN, trained and tested using an independent set of slides with the same methodology to confirm its reproducibility, achieved a mean sensitivity, specificity, and accuracy of 98% (SD = 0), 88% (SD = 0), and 93% (SD = 0), respectively, with an AUC of 0.97. Thus, a high-accuracy classifier can be generated for a specific test environment with few images. Although such classifiers may not yield similar performances on slides from another institution,5Niazi M.K.K. Parwani A.V. Gurcan M.N. Digital pathology and artificial intelligence.Lancet Oncol. 2019; 20: e253-e261Google Scholar the practical application of environment-specific assistance tools may be more realistic than an attempt to achieve broad generalization across all environments. Such systems might be the most beneficial for less experienced pathologists. For experienced pathologists, the systems could provide a triage.5Niazi M.K.K. Parwani A.V. Gurcan M.N. Digital pathology and artificial intelligence.Lancet Oncol. 2019; 20: e253-e261Google Scholar Further studies are required to investigate CNN-based classifiers in a real-life setting. Dr Brinker would like to disclose that he owns a health technology company (Smart Health Heidelberg GmbH; https://smarthealth.de), which develops mobile apps, outside the submitted work. Dr Beltraminelli would like to disclose that he received honoraria for his role on the Takeda Pharma advisory board, outside the submitted work. The other authors have no conflicts of interest to declare. We thank Dr Kenneth S. Resnick for his participation in the survey.

Topics & Concepts