Is Google Gemini better than ChatGPT at evaluating research quality?

Mike Thelwall

2025Journal of Data and Information Science17 citationsDOI Open Access PDF

Abstract

ABSTRACT Google Gemini 1.5 Flash scores were compared with ChatGPT 4o-mini on evaluations of (a) 51 of the author’s journal articles and (b) up to 200 articles in each of 34 field-based Units of Assessment (UoAs) from the UK Research Excellence Framework (REF) 2021. From (a), the results suggest that Gemini 1.5 Flash, unlike ChatGPT 4o-mini, may work better when fed with a PDF or article full text, rather than just the title and abstract. From (b), Gemini 1.5 Flash seems to be marginally less able to predict an article’s research quality (using a departmental quality proxy indicator) than ChatGPT 4o-mini, although the differences are small, and both have similar disciplinary variations in this ability. Averaging multiple runs of Gemini 1.5 Flash improves the scores.

Topics & Concepts

Quality (philosophy)Computer scienceProject managementEngineering managementKnowledge managementData scienceProcess managementWorld Wide WebEngineeringSystems engineeringEpistemologyPhilosophyArtificial Intelligence in Healthcare and EducationMeta-analysis and systematic reviewsRadiomics and Machine Learning in Medical Imaging