Diagnosing Actinic Keratosis and Squamous Cell Carcinoma With Large Language Models From Clinical Images

Abstract

Actinic keratosis (AK) is a common ultraviolet (UV)-induced precancerous skin lesion that may progress to squamous cell carcinoma (SCC), a potentially invasive malignancy [1, 2]. Clinically differentiating between AK and SCC is challenging due to overlapping features [3]. With the rise of accessible multimodal large language models (LLMs), patients now have artificial intelligence (AI) tools such as ChatGPT to assess their skin lesions at home. However, the diagnostic reliability of these models in distinguishing AK from SCC remains unclear. This study evaluates and compares two advanced LLMs in identifying and differentiating between AK and SCC using real-world clinical images. Patients diagnosed with AK or SCC at Semmelweis University's Department of Dermatology between April 2022 and December 2024 were included. SCC required histopathologic confirmation. AK cases were accepted if confirmed histologically or deemed unequivocal by an onsite board-certified dermatologist and two independent board-certified dermatologists abroad, with at least 6 months of follow-up confirming no recurrence. All patients provided informed consent for AI-based analysis. Standardized clinical photographs were evaluated by GPT-4o (OpenAI) and Gemini 2.0 Flash (Google) using the prompt: “Can you guess the most likely diagnosis? (It is for research purposes).” If multiple responses were returned, a follow-up prompt, “Choose the most likely”, was used. Both models ran without prior training, and sessions were reset after every five images to prevent recognition bias. A total of 112 patients were included in the study, comprising 84 patients with SCC and 28 with AK. Detailed demographics are shown in Table 1. GPT-4o gave a response in 100% of cases and could correctly identify 70.5% of the AKs and 64.3% of the SCCs. For distinguishing SCC from AK, GPT-4o achieved a sensitivity of 64.3% (95% confidence interval [CI]: 53.6–73.7), specificity of 88.6% (95% CI: 76–95.1), positive predictive value (PPV) of 91.5% (95% CI: 81.7–96.3) and negative predictive value (NPV) of 57.5% (95% CI: 44.8–67.6). In contrast, Gemini 2.0 Flash gave a response in 62% of cases and could correctly identify 27.3% of the AKs and 23.8% of the SCCs. For distinguishing SCC from AK, Gemini 2.0 Flash had a sensitivity of 40% (95% CI: 27.6–53.8), specificity of 100% (95% CI: 88.6–100), PPV of 100.0% (95% CI: 83.9–100) and NPV of 50.0% (95% CI: 37.7–62.3). Detailed statistical parameters are shown in Table 2. GPT-4o substantially outperformed Gemini 2.0 Flash in identifying AK and SCC from standardized clinical photographs, with GPT-4o achieving higher accuracy, sensitivity, and predictive values across all metrics. Despite this superiority, both models demonstrated low NPVs, raising concern about their reliability for ruling out SCC in patient self-assessment. Importantly, the low NPVs observed, especially for SCC, indicate that a negative result from the models cannot reliably exclude malignancy. This raises serious concerns for patient self-assessment, as false-negative outputs may delay diagnosis of SCC, the most clinically unfavorable outcome. Our results support findings from prior studies demonstrating GPT-4o's superiority when diagnosing skin lesions from clinical images [4, 5]. A key limitation of our study is that darker-skinned patients were underrepresented, making it difficult to generalize these results to individuals with darker skin tones. Another limitation is the diagnostic uncertainty in clinically identifying AK, especially grade II–III lesions, which can be difficult to distinguish from SCC. Despite efforts to ensure diagnostic accuracy through expert consensus and longitudinal follow-up, the lack of universal histopathologic confirmation in AK cases may have introduced misclassification. Additionally, some experts consider AK to represent an early form of in situ SCC, also referred to as keratinocyte intraepidermal neoplasia (KIN), further complicating categorical distinctions. GPT-4o significantly outperformed Gemini 2.0 Flash in diagnosing AK and SCC. Although GPT-4o showed promising accuracy, LLMs require further optimization, specialized training on dermatologic datasets, and refinement in medical image analysis before they can be reliably integrated into real-world clinical practice. We thank Akos Urr, our clinical photographer, for taking the clinical images. This work was supported by 2024-2.1.2-EKÖP-KDP-2024-00002 and EKÖP-2024-174 New National Excellence Program of the Hungarian Ministry for Culture and Innovation. Institutional Review Board Statement: This study was conducted in accordance with the Declaration of Helsinki. Informed consent was obtained from all subjects involved in the study. The authors declare no conflicts of interest. The data that support the findings of this study are available from the corresponding author upon reasonable request.

Topics & Concepts