Comparing human and AI expertise in the academic peer review process: towards a hybrid approach
Shai Farber
Abstract
This study assesses the strengths and limitations of peer reviews of Humanities and Social Sciences manuscripts conducted by human experts as compared to those completed using AI systems, focusing on the potential benefits of the integration of these two approaches in higher education research. Using a comparative design, it analyzed reviews of ten unpublished papers by ten human reviewers and the AI-based system Claude-3 Anthropic. While AI showed advantages in efficiency and consistency, human expertise was essential for contextual understanding and ethical judgment. The findings highlight distinct strengths and limitations of both methods, suggesting a hybrid approach that leverages their complementary skills. The study proposes strategies for integration, including diverse training data, explainable AI, and fostering human-AI collaboration. Although a combined approach could enhance research rigor and impartiality, challenges remain in addressing biases, ensuring transparency, and building trust. This research sets the stage for future efforts to harmonize human and AI capabilities, aiming for a streamlined, robust peer review process that upholds academic integrity and advances knowledge.