Analytical Validation of a Clinical Grade Prognostic and Classification Artificial Intelligence Laboratory Test for Men with Prostate Cancer
Paul Gerrard, Jingbin Zhang, Rikiya Yamashita, Huei–Chung Huang, Sanghita Nag, Sokha Nhek, Joshua Kish, Adam Cole, Nathan Silberman, Trevor J. Royce, Tim Showalter
Abstract
Introduction: This is the first study of which we are aware to describe the analytical validation (AV) of clinical grade artificial intelligence (AI) algorithms for a commercially available prostate cancer test performed on hematoxylin and eosin stained specimens that is not dependent on a priori established molecules or a priori semantically meaningful morphology. Methods: We adapted AV methods used in molecular diagnostics and clinical pathology to two AI biomarkers used in a clinical test for prostate cancer biopsy specimens. The two algorithms included one algorithm with prognostic performance and a second algorithm predictive for treatment benefit from short-term androgen deprivation therapy (ST-ADT). We assessed analytical accuracy, intra-operator reliability, and inter-operator reliability, and biopsy set completeness reliability on two AI algorithms deployed into a clinical laboratory setting. Analytical accuracy was measured using intraclass correlation coefficient (ICC). Reliability studies were assessed using ICC for the prognostic algorithm and percent agreement for the ST-ADT classification algorithm. Results: Analytical accuracy ICC was 0.991 and 0.934 for the prognostic and ST-ADT algorithms, respectively. Intra-operator reliability was 0.981 (ICC) and 100% (percent agreement) for the prognostic and ST-ADT algorithms, respectively. Inter-operator reliability was 0.994 (ICC) and 93.3% (percent agreement) for the prognostic and ST-ADT algorithms, respectively. Biopsy-completeness reliability for one versus three prostate biopsy cores was 0.894 (ICC) and 91.67% (percent agreement) for the prognostic and ST-ADT algorithms respectively. For one versus six cores, reliability was 0.857 (ICC) and 95.00% (percent agreement) for the prognostic and ST-ADT algorithms respectively. Conclusion: This study describes a novel approach to AV of AI algorithms in prostate cancer and applies this approach to two algorithms translated for use as a clinical grade AI-based laboratory test, supporting analytical validity of the test.