Distribution of race and Fitzpatrick skin types in data sets for deep learning in dermatology: A systematic review

Abstract

To the Editor: Advances in deep learning research have produced algorithms to automate image classification and segmentation, performing similar to or better than physicians. These advances have generated ethical questions regarding equitable representation in data sets used to train algorithms. In dermatology, the disproportionate representation of lighter skin types in cutaneous imaging data sets is a well-known issue,1Adamson A.S. Smith A. Machine learning and health care disparities in dermatology.JAMA Dermatol. 2018; 154: 1247-1248https://doi.org/10.1001/jamadermatol.2018.2348Crossref PubMed Scopus (154) Google Scholar and studies have shown that models trained on a homogeneous, monoracial population performed poorly on patients of another race.2Han S.S. Kim M.S. Lim W. Park G.H. Park I. Chang S.E. Classification of the clinical images for benign and malignant cutaneous tumors using a deep learning algorithm.J Invest Dermatol. 2018; 138: 1529-1538https://doi.org/10.1016/j.jid.2018.01.028Abstract Full Text Full Text PDF PubMed Scopus (307) Google Scholar We conducted a literature review to assess the reported rate of demographic information, specifically race and Fitzpatrick skin type (FST), in cutaneous imaging deep learning publications.The following databases were searched for peer-reviewed articles: PubMed, Scopus, and Web of Science. The inclusion criteria comprised retrospective or prospective studies that tested a deep learning algorithm performing a classification or a segmentation task on a skin imaging database and were published online or in print in English before January 20, 2021. Studies were excluded if the article used histopathologic images.Our initial literature search returned 228 unique articles. After filtering based on the inclusion and exclusion criteria, 52 articles remained. Twenty (38.5%) articles tested their models on proprietary data sets from collaborating clinics and academic centers, whereas the remaining 32 (61.5%) sourced their data from a combination of publicly available data sets such as the Pedro Hispanic Hospital and the Human Against Machine with 10000 images (HAM10000) data sets.3Mendonca T. Ferreira P.M. Marques J.S. Marcal A.R. Rozeira J. PH2 - a dermoscopic image database for research and benchmarking.Annu Int Conf IEEE Eng Med Biol Soc. 2013; 2013: 5437-5440https://doi.org/10.1109/EMBC.2013.6610779Crossref PubMed Scopus (506) Google Scholar,4Tschandl P. Rosendahl C. Kittler H. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions.Sci Data. 2018; 5: 180161https://doi.org/10.1038/sdata.2018.161Crossref PubMed Scopus (730) Google ScholarOf the 52 articles, only 9 (17.3%) provided distributions of race and/or FST as summarized in Table I. Four (7.7%) articles presented the distribution of both race and FST, 1 (1.9%) presented only racial distributions, and 4 (7.7%) provided only FST distributions. Of the 9 articles, 5 (55.6%) sourced data from Europe, 2 (22.2%) from Asia, 1 (11.1%) from North America, and 1 (11.1%) from South America. When evaluating the breakdown of FST, the distribution skewed toward lighter skin types. Of a total of approximately 30,000 aggregate images, only 2.1% were type V and type VI. Only 1 of the studies obtained FST information from patient charts, whereas the rest did not specify their source. Limited representation of Black, Latin, and Native American ethnicities was noted. Articles with race or FST information used unique proprietary data sets. While no articles using public databases provided this information, Kinyanjui et al5Kinyanjui NM, Odonga T, Cintas C, et al. Fairness of classifiers across skin tones in dermatology. In: Martel AL, ed. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. Cham: Springer; 2020. Lecture Notes in Computer Science; 12266. https://doi.org/10.1007/978-3-030-59725-2_31Google Scholar previously showed that they tend to skew heavily toward light skin phototypes.Table IDistribution of race and Fitzpatrick skin types across cutaneous imaging data sets for deep learningAggregate sample sizeNo. of studiesRace Asian182,4333 Black10871 Hawaiian/Pacific Islander2241 Latin/Hispanic70441 Native American1421 White61,2304Fitzpatrick skin type I310∗Sum included sample sizes that were aggregated over >1 Fitzpatrick skin type. These aggregate sample sizes were divided by the number of skin types they represent before calculating the sum across all studies to avoid double counting images.6 II4992∗Sum included sample sizes that were aggregated over >1 Fitzpatrick skin type. These aggregate sample sizes were divided by the number of skin types they represent before calculating the sum across all studies to avoid double counting images.7 III8854∗Sum included sample sizes that were aggregated over >1 Fitzpatrick skin type. These aggregate sample sizes were divided by the number of skin types they represent before calculating the sum across all studies to avoid double counting images.7 IV15,677∗Sum included sample sizes that were aggregated over >1 Fitzpatrick skin type. These aggregate sample sizes were divided by the number of skin types they represent before calculating the sum across all studies to avoid double counting images.7 V590∗Sum included sample sizes that were aggregated over >1 Fitzpatrick skin type. These aggregate sample sizes were divided by the number of skin types they represent before calculating the sum across all studies to avoid double counting images.5 VI584∗ Sum included sample sizes that were aggregated over >1 Fitzpatrick skin type. These aggregate sample sizes were divided by the number of skin types they represent before calculating the sum across all studies to avoid double counting images. Open table in a new tab Only a small percentage of cutaneous imaging deep learning publications provide information regarding race and FST, and of the studies that do so, there is limited representation of darker skin types. One study limitation is the potentially flawed assignment of FSTs, as most articles did not consider the patient history and propensity for burns. We hope this motivates investigators to develop cutaneous imaging databases that reflect the diversity of the patient population and to report demographic distributions so that new algorithms can be properly evaluated for bias. Machine learning will likely play a key role in the future of dermatology. Diligence and transparency are needed to prevent new health care disparities in patients with skin of color. To the Editor: Advances in deep learning research have produced algorithms to automate image classification and segmentation, performing similar to or better than physicians. These advances have generated ethical questions regarding equitable representation in data sets used to train algorithms. In dermatology, the disproportionate representation of lighter skin types in cutaneous imaging data sets is a well-known issue,1Adamson A.S. Smith A. Machine learning and health care disparities in dermatology.JAMA Dermatol. 2018; 154: 1247-1248https://doi.org/10.1001/jamadermatol.2018.2348Crossref PubMed Scopus (154) Google Scholar and studies have shown that models trained on a homogeneous, monoracial population performed poorly on patients of another race.2Han S.S. Kim M.S. Lim W. Park G.H. Park I. Chang S.E. Classification of the clinical images for benign and malignant cutaneous tumors using a deep learning algorithm.J Invest Dermatol. 2018; 138: 1529-1538https://doi.org/10.1016/j.jid.2018.01.028Abstract Full Text Full Text PDF PubMed Scopus (307) Google Scholar We conducted a literature review to assess the reported rate of demographic information, specifically race and Fitzpatrick skin type (FST), in cutaneous imaging deep learning publications. The following databases were searched for peer-reviewed articles: PubMed, Scopus, and Web of Science. The inclusion criteria comprised retrospective or prospective studies that tested a deep learning algorithm performing a classification or a segmentation task on a skin imaging database and were published online or in print in English before January 20, 2021. Studies were excluded if the article used histopathologic images. Our initial literature search returned 228 unique articles. After filtering based on the inclusion and exclusion criteria, 52 articles remained. Twenty (38.5%) articles tested their models on proprietary data sets from collaborating clinics and academic centers, whereas the remaining 32 (61.5%) sourced their data from a combination of publicly available data sets such as the Pedro Hispanic Hospital and the Human Against Machine with 10000 images (HAM10000) data sets.3Mendonca T. Ferreira P.M. Marques J.S. Marcal A.R. Rozeira J. PH2 - a dermoscopic image database for research and benchmarking.Annu Int Conf IEEE Eng Med Biol Soc. 2013; 2013: 5437-5440https://doi.org/10.1109/EMBC.2013.6610779Crossref PubMed Scopus (506) Google Scholar,4Tschandl P. Rosendahl C. Kittler H. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions.Sci Data. 2018; 5: 180161https://doi.org/10.1038/sdata.2018.161Crossref PubMed Scopus (730) Google Scholar Of the 52 articles, only 9 (17.3%) provided distributions of race and/or FST as summarized in Table I. Four (7.7%) articles presented the distribution of both race and FST, 1 (1.9%) presented only racial distributions, and 4 (7.7%) provided only FST distributions. Of the 9 articles, 5 (55.6%) sourced data from Europe, 2 (22.2%) from Asia, 1 (11.1%) from North America, and 1 (11.1%) from South America. When evaluating the breakdown of FST, the distribution skewed toward lighter skin types. Of a total of approximately 30,000 aggregate images, only 2.1% were type V and type VI. Only 1 of the studies obtained FST information from patient charts, whereas the rest did not specify their source. Limited representation of Black, Latin, and Native American ethnicities was noted. Articles with race or FST information used unique proprietary data sets. While no articles using public databases provided this information, Kinyanjui et al5Kinyanjui NM, Odonga T, Cintas C, et al. Fairness of classifiers across skin tones in dermatology. In: Martel AL, ed. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. Cham: Springer; 2020. Lecture Notes in Computer Science; 12266. https://doi.org/10.1007/978-3-030-59725-2_31Google Scholar previously showed that they tend to skew heavily toward light skin phototypes. Only a small percentage of cutaneous imaging deep learning publications provide information regarding race and FST, and of the studies that do so, there is limited representation of darker skin types. One study limitation is the potentially flawed assignment of FSTs, as most articles did not consider the patient history and propensity for burns. We hope this motivates investigators to develop cutaneous imaging databases that reflect the diversity of the patient population and to report demographic distributions so that new algorithms can be properly evaluated for bias. Machine learning will likely play a key role in the future of dermatology. Diligence and transparency are needed to prevent new health care disparities in patients with skin of color. None disclosed.

Topics & Concepts