构建多跳推理评估新基准，揭示大模型推理断点。
Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models
- 设计4跳问答数据集，分解每步子问题与中间答案
- 发现模型在后半程推理中易出错且错误会传播
- 可用于提升模型推理能力，适合研究者优化模型
仅通过最终答案评估大语言模型（LLM）的推理能力会掩盖中间步骤的失败，尤其在缺乏步骤标注的多跳问答（multi-hop QA）基准上。为此，我们提出Omanic，一个开放域4跳问答基准，不仅衡量最终答案准确率，还诊断推理过程中的断裂点。Omanic包含10,296个机器生成的训练样本（OmanicSynth）和967个专家评审的人工标注评估样本（OmanicBench），每个问题被分解为单跳子问题、中间答案及结构化图拓扑。对专有和开源LLM的实验表明，Omanic具有挑战性，步骤分析揭示后期跳跃瓶颈、事实知识下限及推理链中的误差传播。在OmanicSynth上微调可迁移至六个推理与数学基准，平均提升7.41分，验证其作为推理能力监督信号的有效性。数据已公开于https://huggingface.co/datasets/li-lab/Omanic，代码见https://github.com/XiaojieGu/Omanic。
原文摘要 · Abstract (English)
Evaluating the reasoning abilities of large language models (LLMs) solely from final answers can obscure failures in intermediate steps, especially in multi-hop QA benchmarks without step-level annotations. To address this gap, we introduce Omanic, an open-domain 4-hop QA benchmark designed not only to measure final-answer accuracy but also to diagnose where reasoning breaks down. Omanic contains 10,296 machine-generated training examples (OmanicSynth) and 967 expert-reviewed human-annotated evaluation examples (OmanicBench), with each evaluation question decomposed into single-hop sub-questions, intermediate answers, and structured graph topologies. Experiments with proprietary and open-source LLMs show that Omanic is challenging, while step-wise analysis reveals a later-hop bottleneck, factual knowledge floor, and error propagation along reasoning chains. Fine-tuning on OmanicSynth transfers to six reasoning and mathematics benchmarks, yielding a 7.41-point average gain and validating its effectiveness as supervision for reasoning-capability transfer. We release the data at https://huggingface.co/datasets/li-lab/Omanic and the code at https://github.com/XiaojieGu/Omanic.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。