TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Models
Jae‐Woo Ahn, Taehyun Lee, Junyoung Lim, Jin-Hwa Kim, Sangdoo Yun, Hwaran Lee, Gunhee Kim
Abstract
While Large Language Models (LLMs) can serve as agents to simulate human behaviors (i.e., role-playing agents), we emphasize the importance of point-in-time role-playing.This situates characters at specific moments in the narrative progression for three main reasons: (i) enhancing users' narrative immersion, (ii) avoiding spoilers, and (iii) fostering engagement in fandom role-playing.To accurately represent characters at specific time points, agents must avoid character hallucination, where they display knowledge that contradicts their characters' identities and historical timelines.We introduce TIMECHARA, a new benchmark designed to evaluate point-in-time character hallucination in role-playing LLMs.Comprising 10,895 instances generated through an automated pipeline, this benchmark reveals significant hallucination issues in current state-ofthe-art LLMs (e.g., GPT-4o).To counter this challenge, we propose NARRATIVE-EXPERTS, a method that decomposes the reasoning steps and utilizes narrative experts to reduce point-intime character hallucinations effectively.Still, our findings with TIMECHARA highlight the ongoing challenges of point-in-time character hallucination, calling for further study. 1 Related WorkWe include a more thorough literature review in Appendix A. In this section, we only discuss the most relevant works.