Contrived Materials and a Data Set for the Evaluation of Liquid Biopsy Tests

Abstract

The Blood Profiling Atlas in Cancer (BLOODPAC) Consortium is a collaborative effort involving stakeholders from the public, industry, academia, and regulatory agencies focused on developing shared best practices on liquid biopsy. This report describes the results from the JFDI (Just Freaking Do It) study, a BLOODPAC initiative to develop standards on the use of contrived materials mimicking cell-free circulating tumor DNA, to comparatively evaluate clinical laboratory testing procedures. Nine independent laboratories tested the concordance, sensitivity, and specificity of commercially available contrived materials with known variant-allele frequencies (VAFs) ranging from 0.1% to 5.0%. Each participating laboratory utilized its own proprietary evaluation procedures. The results demonstrated high levels of concordance and sensitivity at VAFs of >0.1%, but reduced concordance and sensitivity at a VAF of 0.1%; these findings were similar to those from previous studies, suggesting that commercially available contrived materials can support the evaluation of testing procedures across multiple technologies. Such materials may enable more objective comparisons of results on materials formulated in-house at each center in multicenter trials. A unique goal of the collaborative effort was to develop a data resource, the BLOODPAC Data Commons, now available to the liquid-biopsy community for further study. This resource can be used to support independent evaluations of results, data extension through data integration and new studies, and retrospective evaluation of data collection. The Blood Profiling Atlas in Cancer (BLOODPAC) Consortium is a collaborative effort involving stakeholders from the public, industry, academia, and regulatory agencies focused on developing shared best practices on liquid biopsy. This report describes the results from the JFDI (Just Freaking Do It) study, a BLOODPAC initiative to develop standards on the use of contrived materials mimicking cell-free circulating tumor DNA, to comparatively evaluate clinical laboratory testing procedures. Nine independent laboratories tested the concordance, sensitivity, and specificity of commercially available contrived materials with known variant-allele frequencies (VAFs) ranging from 0.1% to 5.0%. Each participating laboratory utilized its own proprietary evaluation procedures. The results demonstrated high levels of concordance and sensitivity at VAFs of >0.1%, but reduced concordance and sensitivity at a VAF of 0.1%; these findings were similar to those from previous studies, suggesting that commercially available contrived materials can support the evaluation of testing procedures across multiple technologies. Such materials may enable more objective comparisons of results on materials formulated in-house at each center in multicenter trials. A unique goal of the collaborative effort was to develop a data resource, the BLOODPAC Data Commons, now available to the liquid-biopsy community for further study. This resource can be used to support independent evaluations of results, data extension through data integration and new studies, and retrospective evaluation of data collection. Circulating tumor (ct)-DNA holds great promise in early cancer detection; the identification of therapeutic targets, especially when tissue is not available; the evaluation of residual disease after curative-intent treatment; therapeutic monitoring; as well as resistance mapping and recurrence surveillance [US Food and Drug Administration (FDA), https://www.fda.gov/drugs/resources-information-approved-drugs/cobas-egfr-mutation-test-v2; and Foundation Medicine, https://www.foundationmedicine.com/press-releases/af7bb7df-2dcf-411f-bc7d-ebb8ab90d788, both last accessed January 12, 2023].1Diehl F. Schmidt K. Choti M.A. Romans K. Goodman S. Li M. Thornton K. Agrawal N. Sokoll L. Szabo S.A. Kinzler K.W. Vogelstein B. Diaz Jr., L.A. Circulating mutant DNA to assess tumor dynamics.Nat Med. 2008; 14: 985-990Crossref PubMed Scopus (1999) Google Scholar, 2Kwapisz D. The first liquid biopsy test approved. Is it a new era of mutation testing for non-small cell lung cancer?.Ann Transl Med. 2017; 5: 1-7Crossref Scopus (188) Google Scholar, 3Keppens C. Palma J.F. Das P.M. Scudder S. Wen W. Normanno N. van Krieken J.H. Sacco A. Fenizia F. Gonzalez de Castro D. Hönigschnabl S. Kern I. Lopez-Rios F. Lozano M.D. Marchetti A. Halfon P. Schuuring E. Setinek U. Sorensen B. Taniere P. Tiemann M. Vosmikova H. Dequeker E.M.C. Detection of EGFR variants in plasma: a multilaboratory comparison of a real-time PCR EGFR mutation test in Europe.J Mol Diagn. 2018; 20: 483-494Abstract Full Text Full Text PDF PubMed Scopus (34) Google Scholar, 4Weber S. Spiegl B. Perakis S.O. Ulz C.M. Abuja P.M. Kashofer K. van der Leest P. Azpurua M.A. Tamminga M. Brudzewsky D. Rothwell D.G. Mohan S. Sartori A. Lampignano R. Konigshofer Y. Sprenger-Haussels M. Wikman H. Bergheim I.R. Kloten V. Schuuring E. Speicher M.R. Heitzer E. Technical evaluation of commercial mutation analysis platforms and reference materials for liquid biopsy profiling.Cancers (Basel). 2020; 12: 1-16Crossref Scopus (30) Google Scholar For example, early work by Diehl et al1Diehl F. Schmidt K. Choti M.A. Romans K. Goodman S. Li M. Thornton K. Agrawal N. Sokoll L. Szabo S.A. Kinzler K.W. Vogelstein B. Diaz Jr., L.A. Circulating mutant DNA to assess tumor dynamics.Nat Med. 2008; 14: 985-990Crossref PubMed Scopus (1999) Google Scholar demonstrated that ctDNA could be used to monitor tumor dynamics in colorectal cancer patients undergoing surgery or chemotherapy, thus revealing its potential as a clinically useful biomarker. The oncology field has utilized multiple methods of detecting ctDNA, such as Droplet Digital PCR (ddPCR) and, more recently, massively parallel next-generation sequencing (NGS). ctDNA is present at very low levels against a background of wild-type (WT) cell-free (cf)-DNA that is derived from various origins. The FDA has approved the use of several ctDNA tests, including a qPCR test for use in the identification of epidermal growth factor receptor gene (EGFR) mutations in non–small-cell lung cancer patients,2Kwapisz D. The first liquid biopsy test approved. Is it a new era of mutation testing for non-small cell lung cancer?.Ann Transl Med. 2017; 5: 1-7Crossref Scopus (188) Google Scholar,3Keppens C. Palma J.F. Das P.M. Scudder S. Wen W. Normanno N. van Krieken J.H. Sacco A. Fenizia F. Gonzalez de Castro D. Hönigschnabl S. Kern I. Lopez-Rios F. Lozano M.D. Marchetti A. Halfon P. Schuuring E. Setinek U. Sorensen B. Taniere P. Tiemann M. Vosmikova H. Dequeker E.M.C. Detection of EGFR variants in plasma: a multilaboratory comparison of a real-time PCR EGFR mutation test in Europe.J Mol Diagn. 2018; 20: 483-494Abstract Full Text Full Text PDF PubMed Scopus (34) Google Scholar as well as two NGS-based tests for use in detecting various genomic aberrations for therapeutic indications across a variety of solid tumors (FDA, https://www.fda.gov/drugs/resources-information-approved-drugs/cobas-egfr-mutation-test-v2; and Foundation Medicine, https://www.foundationmedicine.com/press-releases/af7bb7df-2dcf-411f-bc7d-ebb8ab90d788, both last accessed January 12, 2023). As novel ctDNA-based methods are advanced toward use in clinical practice to help inform decisions beyond targeted-therapy selection and tumor profiling in late-stage patients, rigorous demonstration of analytical performance continues to be essential in supporting the suitability of tests for their intended uses. Crucial to the adoption of liquid-biopsy methods is the demonstration of performance characteristics such as precision, accuracy, and reproducibility.5Addona T.A. Abbatiello S.E. Schilling B. Skates S.J. Mani D.R. Bunk D.M. et al.Multi-site assessment of the precision and reproducibility.Nat Biotechnol. 2010; 27: 633-641Crossref Scopus (882) Google Scholar, 6Williams P.M. Forbes T. Lund S P. Cole K.D. He H.-J. Karlovich C. Paweletz C.P. Stetson D. Yee L.M. Connors D.E. Keating S.M. Destenaves B. Cleveland M.H. Lau C.J. Barrett J.C. Kelloff G.J. McCormack R.T. Validation of ctDNA quality control materials through a precompetitive collaboration of the Foundation for the National Institutes of Health.J Clin Oncol Precis Oncol. 2021; 5: 910-920Google Scholar, 7Godsey J.H. Silvestro A. Barrett J.C. Bramlett K. Chudova D. Deras I. Dickey J. Hicks J. Johann D.J. Leary R. Lee J.S.H. McMullen J. McShane L. Nakamura K. Richardson A.O. Ryder M. Simmons J. Tanzella K. Yee L. Leiman L.C. Generic protocols for the analytical validation of next-generation sequencing-based ctDNA assays: a joint consensus recommendation of the BloodPAC's analytical variables working group.Clin Chem. 2020; 66: 1156-1166Crossref PubMed Scopus (45) Google Scholar, 8Torga G. Pienta K.J. Patient-Paired sample congruence between 2 commercial liquid biopsy tests.JAMA Oncol. 2018; 4: 868-870Crossref PubMed Scopus (142) Google Scholar, 9Deveson I.W. Gong B. Lai K. LoCoco J.S. Richmond T.A. Schageman J. et the analytical of circulating tumor DNA sequencing for precision Biotechnol. 2021; PubMed Scopus Google Scholar are to develop including those by the Foundation for the National Institutes of to contrived materials in support of and by the FDA 2 to develop standards for ctDNA P.M. Forbes T. Lund S P. Cole K.D. He H.-J. Karlovich C. Paweletz C.P. Stetson D. Yee L.M. Connors D.E. Keating S.M. Destenaves B. Cleveland M.H. Lau C.J. Barrett J.C. Kelloff G.J. McCormack R.T. Validation of ctDNA quality control materials through a precompetitive collaboration of the Foundation for the National Institutes of Health.J Clin Oncol Precis Oncol. 2021; 5: 910-920Google D. R. B. W. Johann D.J. J. quality control of sequencing for precision Cancer Med. 2021; 4: Google Scholar ctDNA tests to performance for testing of J.H. Silvestro A. Barrett J.C. Bramlett K. Chudova D. Deras I. Dickey J. Hicks J. Johann D.J. Leary R. Lee J.S.H. McMullen J. McShane L. Nakamura K. Richardson A.O. Ryder M. Simmons J. Tanzella K. Yee L. Leiman L.C. Generic protocols for the analytical validation of next-generation sequencing-based ctDNA assays: a joint consensus recommendation of the BloodPAC's analytical variables working group.Clin Chem. 2020; 66: 1156-1166Crossref PubMed Scopus (45) Google results in could to of mutations at low as well as of in such as and et P.M. Forbes T. Lund S P. Cole K.D. He H.-J. Karlovich C. Paweletz C.P. Stetson D. Yee L.M. Connors D.E. Keating S.M. Destenaves B. Cleveland M.H. Lau C.J. Barrett J.C. Kelloff G.J. McCormack R.T. Validation of ctDNA quality control materials through a precompetitive collaboration of the Foundation for the National Institutes of Health.J Clin Oncol Precis Oncol. 2021; 5: 910-920Google Scholar the in detecting and in when reference those were of the of the study, et P.M. Forbes T. Lund S P. Cole K.D. He H.-J. Karlovich C. Paweletz C.P. Stetson D. Yee L.M. Connors D.E. Keating S.M. Destenaves B. Cleveland M.H. Lau C.J. Barrett J.C. Kelloff G.J. McCormack R.T. Validation of ctDNA quality control materials through a precompetitive collaboration of the Foundation for the National Institutes of Health.J Clin Oncol Precis Oncol. 2021; 5: 910-920Google Scholar of variants at low The findings from those the for and methods of ctDNA tests and, materials are in may not a or of for performance and be accessed by of the community at these a community collaboration across Blood Profiling Atlas in Cancer (BLOODPAC) was with the goal of the quality and of ctDNA analysis through testing of commercially available reference materials to a data that can as the for and objective BLOODPAC stakeholders across industry, and to on that to cancer B. S. Barrett J.C. D. Bramlett K. et to profiling Atlas in cancer 2017; PubMed Scopus Google Scholar the BLOODPAC Data data resource now available to the liquid-biopsy community for further S.E. Johann D.J. A. Lee J.S.H. C. L. W. Leiman L. data for liquid biopsy Clin Oncol Clin Cancer 2021; 5: Scholar and J.H. Silvestro A. Barrett J.C. Bramlett K. Chudova D. Deras I. Dickey J. Hicks J. Johann D.J. Leary R. Lee J.S.H. McMullen J. McShane L. Nakamura K. Richardson A.O. Ryder M. Simmons J. Tanzella K. Yee L. Leiman L.C. Generic protocols for the analytical validation of next-generation sequencing-based ctDNA assays: a joint consensus recommendation of the BloodPAC's analytical variables working group.Clin Chem. 2020; 66: 1156-1166Crossref PubMed Scopus (45) Google Barrett J.C. et data for liquid biopsy data to 2020; Scopus Google D. J. J. M. C. Keating S. Kelloff G. Leiman L. McCormack R. D. E. K. C. A. T. C. M. liquid biopsy Oncol 2020; PubMed Scopus Google Scholar such as the JFDI (Just Freaking Do It) a multicenter of ctDNA commercially available reference This report the results from the JFDI independent laboratories used ctDNA testing methods and proprietary evaluation procedures to assess the performance of two commercially available reference in or a to were present in a across a of VAFs to The of and concordance, sensitivity, and laboratories utilized sequencing and analysis to comparisons it is could be a data This utilized commercially available contrived reference the and ctDNA The as reference or in is a of each of in at a of The is by genomic DNA derived from a proprietary variants and at and VAF as well as a VAF (WT) for specificity assessment A of to was by the and the DNA The VAFs of variants were by A of to was with the to that reference is derived from cell sequencing data variants across the that were present in the background of the of DNA by in the at and variants are available from available from last last accessed and the of in Cancer last accessed DNA Circulating Droplet Digital in a new variants are available from Data available from last last accessed and the of in Cancer last accessed DNA Circulating Droplet Digital The ctDNA and reference materials for the evaluation of DNA variants a background of genomic DNA derived from cell for at VAFs of and (WT) the were variants and were beyond the of thus were from The DNA was at a between and a The ctDNA was formulated in a with a DNA of in DNA of and The ctDNA was formulated in a and at a of and at Digital were to DNA of DNA in the ctDNA and and to are variants and were not in study, was focused on variants and available from last last accessed and the of in Cancer last accessed DNA ctDNA, Circulating Droplet Digital not in a new to are variants and were not in study, was focused on variants and Data available from last last accessed and the of in Cancer last accessed DNA ctDNA, Circulating Droplet Digital not Each commercial reference a of variants and with EGFR variants and shared between both to the of study, comparisons were between reference materials from each and DNA in and were to Each their or and sequencing or or Droplet Digital PCR in and at a of DNA DNA from to used or that more those present in the contrived materials and that not of those present in the contrived materials with and and of Droplet Digital not next-generation of for in a new Droplet Digital not next-generation of for their own in-house for each and a of and variants not by their sequencing the by the are in The and VAFs were used to a was as or in not by were as not The of control was used as the to and resistance including DNA and was in with the The were on a A of were as in of for The analysis was on the a that not on unique but a with a background to sensitivity and N. A. L. P. C. C. to in 2018; Scopus Google Scholar a PCR to and the of each a to the DNA of in a derived from the that the or DNA was from those that not each was and a of of and VAFs in the sample were of control DNA, for and Detection at of and in a was The used and EGFR A of of and of were a Droplet the Droplet was and through a to A of of the and sample were a and in a at for for and for and for The was to a Droplet and by VAF was and to BLOODPAC for in the JFDI data and The Circulating was used for the of the with a of and were in the of to with the for the the to was and of were DNA from the were were from the For a with was by two and PCR to were from the to the were as and on Data were a analysis For in was the sequencing were the to the with of contrived as were the and quality was on the were to and was the on the and on the were sequencing was in independent The were the Circulating to the The was in of the The of the and the were on the the to the A in the of to was used to the the ctDNA The of the ctDNA of DNA for a DNA of to on the for study, were from of to be with The ctDNA is that and tumor The were on the were as a on cell in the The ctDNA was used for data analysis and For reference analysis of the was The was on the with 2 of The was a of the were with sample unique and the were for a the were and real-time were on Data were a analysis involving to the specificity of high sensitivity of The contrived were used as a to focused on and resistance The were on were at a of at the of in a of for each with The is that unique to of DNA, for high sensitivity and specificity of variants and from A proprietary data analysis and for was used for For reference materials in a DNA was from the the Circulating A of of contrived DNA were used as for to the of DNA in of The of was by qPCR to The data by were derived from of contrived from and For the contrived that is in a 2 or of the control was the cell-free the and for a of of the control materials were for a and of the contrived was used as the were with the to and resistance were the in the The is that unique to of DNA that are and a for sequencing on the sequencing for high sensitivity and specificity of variants and from The were on the for were as a on the or as a on the with a of to of each or gene of for the and were used for data analysis and analysis for sequencing were the to the with of contrived as were the and and quality was the and DNA were to the for with the and and on the data were and The was used with for The sensitivity of each unique was as is the of variants as and is the of variants as those not by the was as the reference is the of that to variants in the reference were as or mutation and is the of that variants present in the reference were in the for each were were both and across concordance data were first by reference contrived and VAF each the of each was variants not by that comparisons were used to the concordance as is the of variants between two and is the of variants in A and is the of variants in B. For concordance, a similar was but of each from a was with from to the of variants by both The used in is available from the BLOODPAC Data for is available on last accessed A effort was to the of contrived reference materials in testing of ctDNA and across a of available contrived reference materials from two were to multiple the BLOODPAC A of independent reference materials across VAF from DNA or used used the NGS-based used sequencing platforms and used the two used sequencing This was focused on variants from the materials and variants from the of variants were shared across both of reference the on each utilized in across the and The a of of was to the in and was from the results of after The of VAFs across NGS-based concordance between and VAFs with both and 2 results, with for that not results on each factor and and VAFs between DNA and the data was first reduced to tested both and variants that were by at of these results similar VAFs between the DNA and the reference in VAFs between the DNA and across VAF from of the reference materials VAFs by by the of DNA VAF and reference with data from both DNA and and variants by at across DNA were The VAF results, VAF the and VAF VAF were between DNA and at the 0.1% was Circulating DNA a between DNA and across the VAF Circulating DNA was by first the sensitivity and specificity of each The of data across reference DNA and VAF specificity was high across of DNA sensitivity between was of reduced sensitivity at the 0.1% VAF The in the high sensitivity at 0.1% to with low specificity in to the of that used the the sample was low to these findings a to both and concordance of variants were as a for concordance was reduced at the 0.1% VAF and concordance were concordance The of BLOODPAC and the Cancer last accessed January 12, between and the of data and results across both and commercial with those was of a BLOODPAC in the was and for of The from of tests, to developing new data and The were including the with proprietary results and the of the of and in the of by the was As the to the of a the of the to as a for the of contrived reference multiple methods were the was not to a to the best test but to evaluate available reference materials and the data with the liquid-biopsy from sequencing data were by each of the participating to the BLOODPAC Data last accessed are The data analysis was by various the working the for the liquid-biopsy community from of was the evaluation of contrived sample by two of the standards and through the use of sequencing contrived of genomic DNA, to DNA a variety of and at and as both as well as The in variants in the two contrived a but on of the the sensitivity and of the used for mutation were and on the VAF to the by the The of is that the sensitivity levels of the contrived from and are at VAF levels across DNA and the findings from that the contrived reference materials across independent to assess and two for sample results support the use of these reference materials in the and validation of liquid-biopsy The findings the of et P.M. Forbes T. Lund S P. Cole K.D. He H.-J. Karlovich C. Paweletz C.P. Stetson D. Yee L.M. Connors D.E. Keating S.M. Destenaves B. Cleveland M.H. Lau C.J. Barrett J.C. Kelloff G.J. McCormack R.T. Validation of ctDNA quality control materials through a precompetitive collaboration of the Foundation for the National Institutes of Health.J Clin Oncol Precis Oncol. 2021; 5: 910-920Google that the reference materials are a resource in the validation of materials can help to the accuracy, and of a for variants from when clinical are very and are used to and the for to support rigorous Circulating from cancer patients is a of and genomic The of on the of in the in on the of available from the and the The tests for were of DNA, to at a a of be with from to of control is more at 0.1% the for testing is to be to or are to be available for to 0.1% is in the of results in a of at low variables such as and mutations present in can further to the of the these potential of were not these in D. J. J. M. C. Keating S. Kelloff G. Leiman L. McCormack R. D. E. K. C. A. T. C. M. liquid biopsy Oncol 2020; PubMed Scopus Google Scholar The findings from were to those from a by the P.M. Forbes T. Lund S P. Cole K.D. He H.-J. Karlovich C. Paweletz C.P. Stetson D. Yee L.M. Connors D.E. Keating S.M. Destenaves B. Cleveland M.H. Lau C.J. Barrett J.C. Kelloff G.J. McCormack R.T. Validation of ctDNA quality control materials through a precompetitive collaboration of the Foundation for the National Institutes of Health.J Clin Oncol Precis Oncol. 2021; 5: 910-920Google Scholar with a similar goal of the of contrived materials to liquid biopsy. between the and the is on the reference standards The present reference standards that are commercially the with new reference materials that were unique to the and not commercially available at the of was a of in the of JFDI BLOODPAC and the study, and the results were shared and between the two as the testing the data from the is not available to the public, of the data from the present is available through the BLOODPAC Data The Cancer initiative the of BLOODPAC in with the goal of and in the liquid-biopsy The BLOODPAC to to and data available through the BLOODPAC Data Commons, a that the as the National Cancer Cancer Data to data and analysis The and that The of these data at the liquid-biopsy community both on best including data Barrett J.C. et data for liquid biopsy data to 2020; Scopus Google Scholar and analytical J.H. Silvestro A. Barrett J.C. Bramlett K. Chudova D. Deras I. Dickey J. Hicks J. Johann D.J. Leary R. Lee J.S.H. McMullen J. McShane L. Nakamura K. Richardson A.O. Ryder M. Simmons J. Tanzella K. Yee L. Leiman L.C. Generic protocols for the analytical validation of next-generation sequencing-based ctDNA assays: a joint consensus recommendation of the BloodPAC's analytical variables working group.Clin Chem. 2020; 66: 1156-1166Crossref PubMed Scopus (45) Google Scholar and on available D. J. J. M. C. Keating S. Kelloff G. Leiman L. McCormack R. D. E. K. C. A. T. C. M. liquid biopsy Oncol 2020; PubMed Scopus Google Scholar of the in be when the or analysis procedures were not but were to use the methods in their for the evaluation of cell-free from were to use and to use their for each of the results of was the a such as that commercial can the independent evaluation similar results across and the for multicenter The data support independent evaluation and extension that has the potential for new The results support testing of clinical similar to those could be used to of these commercially available reference materials with This comparison may from more across PCR platforms and potential and Food and Drug for their and on BLOODPAC the BLOODPAC G. and L. of and of and Cancer and Bramlett of Chudova Dickey and H. Lee and J. for and of and I. and at Cancer and BLOODPAC Consortium to the JFDI including M. and and and and and the and and and and and the approved the and are for the of with with

Topics & Concepts