Evaluating Step-by-step Reasoning Traces: A Survey
Jinu Lee, Julia Hockenmaier
Abstract
Step-by-step reasoning is widely used to enhance the reasoning ability of large language models (LLMs) in complex problems.Evaluating the quality of reasoning traces is crucial for understanding and improving LLM reasoning.However, existing evaluation practices are highly inconsistent, resulting in fragmented progress across evaluator design and benchmark development.To address this gap, this survey provides a comprehensive overview of step-by-step reasoning evaluation, proposing a taxonomy of evaluation criteria with four toplevel categories (factuality, validity, coherence, and utility).Based on the taxonomy, we review different datasets, evaluator implementations, and recent findings, leading to promising directions for future research. QueryTrace 2Trace 1Trace 3Meta-evaluation benchmarks:Can evaluator classify good/bad steps?Verifier-guided search:Can the evaluator choose the most promising trace?Reinforcement learning: Are evaluator scores a good reward function? Chosen trace answer: Correct Performance of trained LLM Classification accuracy