Artificial intelligence recognition of cutaneous chronic <scp>graft‐versus‐host</scp> disease by a deep learning neural network

Abstract

Chronic graft-versus-host disease (cGVHD) is the leading cause of non-relapse long-term morbidity and mortality in patients after allogeneic haematopoietic cell transplantation (HCT).1 Skin is the earliest and most commonly affected organ and has a central role in evaluating treatment efficacy and disease progression.2 By way of National Institutes of Health (NIH) scoring, the affected cutaneous body surface area (BSA) has been incorporated into study design to test all three current FDA-approved cGVHD treatments. For example, one inclusion criterion for the pivotal trial of ibrutinib was a minimum of 25% BSA erythema.3 However, visual assessment of cGVHD suffers low reliability, limiting the ability to effectively follow patient response. The gap of measuring affected BSA in known cGVHD patients could be addressed by automated image analysis techniques leveraging artificial intelligence (AI). To this end, we trained a deep learning neural network4 to mark (segment) affected skin and tested performance in 36 previously unseen patients by leave-one-patient-out validation. We further benchmarked the AI against exact human measurements of affected BSA in 3D photographs. Thirty-six patients (characteristics in Table S1) were photographed with a Vectra H1 3D camera (Canfield Scientific, Inc.), for a dataset of 360 three-dimensional cross-polarized photographs. The camera uses ranging lights to ensure precise measurement of the square centimetres of surface captured, and a consistent field of view for each acquired photograph. When the entire body is photographed, the sum of all skin surface areas from each photo equals the total BSA. As human ground truth, areas affected by cGVHD in each photo were marked by an annotator (KP) with >6 months of cGVHD training (Methods: Ground truth annotations in Data S1). 179 photos from 25 patients were marked as having skin affected by cutaneous cGVHD. The remaining 181 photos had unaffected skin (Table S2). We trained a deep learning U-Net algorithm4 to identify skin affected by cGVHD in patient photos (Methods: Algorithm development and model training in Data S1). For leave-one-out validation, each of 36 models was trained on the ground truth annotations from 35 patients. The performance of each model was tested on photos of the unseen patient. Figure 1 shows representative algorithm output. In identifying whether affected skin was present in a photograph, the algorithm achieved an accuracy of 0.92, positive predictive value of 0.92, and negative predictive value of 0.92 at the image level (Table S2). Segmentation performance was only assessed in the 179 photographs of affected skin (by human ground truth). The algorithm achieved a median Dice coefficient of 0.74 (interquartile range: 0.40–0.89) and median surface area error of 8.89% (22.10%–3.69%) (Figure 2A). Performance varied substantially between patients (Figure 2B) and was significantly lower in Fitzpatrick skin types IV–VI, underrepresented in our study (Figure 2C). Whereas the algorithm's median surface area error was under 9% overall, it was 24% for these darker skin types, likely due to the paucity of training examples. To contextualize, Mitchell et al. reported a minimum detectable change for clinicians approximately 20%.5 Disease type showed no consistent effect on the per-patient algorithm performance (Figure S2). Algorithm output was scored as clinically acceptable or better (3+) in 77% of photos, at expert level (5/5) in 20% of photos (Figure 2D), and noninferior to human annotations in 52% of photos (Figure 2E). The mean score for algorithm contours was 3.45 compared to 4.17 for human contours. The median difference of the two scores for an individual photo was 0 (interquartile range −1 to 0) (Figure S1). Thus, most annotations by the algorithm were equivalently acceptable to the ground truth human annotation. Our calibrated 3D photography enabled exact skin surface area in square centimetres to be calculated for each photograph, along with the area marked by the human annotator. To estimate the reliability of the algorithm in predicting affected BSA, we compared the proportion of total skin surface marked by the human for patients with at least three photos (n = 31) to the percentage of skin pixels marked by the algorithm for each patient. Bland–Altman analysis revealed algorithm error comparable to the 20% error reported for clinicians5 (Figure 2F). At the patient level, the algorithm's median absolute BSA error was 2.74% (0.13%-7.29%), mean 5.65% (standard deviation 7.44%). Across four trials, the corresponding mean absolute BSA error for trained NIH scorers estimating extent of epidermal disease (erythema) ranged from 10.42% (4.25%) to 21.72% (9.60%) of BSA.5 We created an automated algorithm that accurately recognized and segmented cGVHD-affected skin. Despite the relatively small dataset, algorithm predictions were assessed by an independent dermatologist as clinically acceptable or better for most patients. Our results suggest that AI could reliably assess clinically affected cGVHD surface area. As this is central to NIH scoring,6 AI holds promise for facilitating reliable cGVHD staging. However, our training photos did not adequately cover the spectrum of disease appearance and suffered significantly lower performance in under-represented darker skin types, reflecting the limited patients available at recruitment. In furthering this approach, the number of patients in underrepresented groups must be expanded to improve AI training and address this performance shortfall. Future studies must assess the clinical utility of AI in practice, ability to track surface area changes over time, and the potential for capturing early changes that are key to enabling early intervention.7 This work was supported by a Career Development Award to Eric Tkaczyk from the United States Department of Veterans Affairs Clinical Sciences R&D Service (IK2 CX001785), a Discovery Research Grant from Vanderbilt University, the Vanderbilt University Medical Center Departments of Medicine and Dermatology, the National Institutes of Health (K12 CA090625, R21 AR074589), and the European Regional Development Fund (1.1.1.2/VIAA/4/20/665). Lee Wheless is funded by grants from The Skin Cancer Foundation and The Dermatology Foundation. We thank Gideon Amos, Dennis O'Neal, and Dennis DaSilva and colleageus at Canfield Scientific for technical assistance with the Vectra 3D photography software. The authors state no conflict of interest. Conceptualization: Andrew McNeil, Eric R. Tkaczyk, Benoit M. Dawant. Data curation: Kelsey Parks, Fuyao Chen, Andrew McNeil, Tahsin Reasat, Eric R. Tkaczyk, Austin Cronin. Formal analysis: Andrew McNeil, Xiaoqi Liu, Austin Cronin. Funding acquisition: Eric R. Tkaczyk, Benoit M. Dawant. Investigation: Andrew McNeil, Eric R. Tkaczyk, Kelsey Parks, Fuyao Chen, Lee Wheless. Methodology: Andrew McNeil, Benoit M. Dawant, Eric R. Tkaczyk. Software: Andrew McNeil, Xiaoqi Liu. Writing – original draft: Andrew McNeil, Inga Saknite. Writing – review & editing: Andrew McNeil, Inga Saknite, Eric R. Tkaczyk, Benoit M. Dawant, Lee Wheless. DataS1 Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.

Topics & Concepts