Automated Mass Extraction of Over 680,000 PICOs from Clinical Study Abstracts Using Generative AI: A Proof-of-Concept Study

Abstract

Generative artificial intelligence (GenAI) shows promise in automating key tasks involved in conducting systematic literature reviews (SLRs), including screening, bias assessment and data extraction. This potential automation is increasingly relevant as pharmaceutical developers face challenging requirements for timely and precise SLRs using the population, intervention, comparator and outcome (PICO) framework, such as those under the impending European Union (EU) Health Technology Assessment Regulation 2021/2282 (HTAR). This proof-of-concept study aimed to evaluate the feasibility, accuracy and efficiency of using GenAI for mass extraction of PICOs from PubMed abstracts. Abstracts were retrieved from PubMed using a search string targeting randomised controlled trials. A PubMed clinical study ‘specific/narrow’ filter was also applied. Retrieved abstracts were processed using the OpenAI Batch application programming interface (API), which allowed parallel processing and interaction with Generative Pre-trained Transformer 4 Omni (GPT-4o) via custom Python scripts. PICO elements were extracted using a zero-shot prompting strategy. Results were stored in CSV files and subsequently imported into a PostgreSQL database. The PubMed search returned 682,667 abstracts. PICOs from all abstracts were extracted in < 3 h, with an average processing time of 200 s per 1000 abstracts. A total of 395,992,770 tokens were processed, with an average of 580 tokens per abstract. The total cost was $3390. On the basis of a random sample of 350 abstracts, human verification confirmed that GPT-4o accurately and comprehensively extracted 342 (98%) of all PICOs, with only outcome elements rarely missed. Using GenAI to extract PICOs from clinical study abstracts could fundamentally transform the way SLRs are conducted. By enabling pharmaceutical developers to anticipate PICO requirements, this approach allows for proactive preparation for the EU HTAR process, or other health technology assessments (HTAs), streamlining efficiency and reducing the burden of meeting these requirements. This study explored how artificial intelligence (AI) can help automate a key part of systematic literature reviews (SLRs), which are used to summarise research in healthcare. AI was used to extract specific information referred to as population, intervention, comparator and outcome (PICOs) from nearly 700,000 abstracts of clinical studies from the PubMed database. AI extracted the PICO information from all the abstracts in under 3 h, which is significantly faster than a human could do. This demonstrates that AI could save a lot of time and effort compared with using human reviewers to do the same task. The study also found that AI performed accurately in identifying the PICO elements in the abstracts, although further human verification is still required. The use of AI in this way could help pharmaceutical developers meet upcoming requirements from the European Union, which require timely and thorough reviews of clinical evidence. AI can speed up the process, reducing the burden on pharmaceutical developers and allowing them to prepare more efficiently for these assessments. However, further testing and validation is needed before using AI in this way becomes common.

Topics & Concepts