首个评估韩语多步软推理的基准，助力大模型理解长篇韩语文本。
Ko-MuSR: A Multistep Soft Reasoning Benchmark for LLMs Capable of Understanding Korean
- 基于韩语文本构建多步推理链与人工验证题库，避免数据污染。
- 多语言模型在韩语推理任务中表现优于专用模型，体现跨语言推理泛化能力。
- 提示工程优化可逼近人类水平，适合研究韩语长文本理解的学者。
我们提出Ko-MuSR，首个全面评估长篇韩语文本中多步软推理能力的基准，同时最大限度减少数据污染。该基准遵循MuSR设计，包含全韩语叙事、推理链及经人工标注验证逻辑一致性与可答性的选择题。对四种大语言模型（两种多语言、两种韩语专用）的评估显示，多语言模型在韩语推理任务中表现更优，表明推理能力具有跨语言泛化性。精心设计的提示策略——结合少样本示例、推理轨迹与任务提示——进一步提升准确率，接近人类水平。Ko-MuSR为推进韩语自然语言处理提供了系统评估长上下文推理与提示策略的坚实基础。
原文摘要 · Abstract (English)
We present Ko-MuSR, the first benchmark to comprehensively evaluate multistep, soft reasoning in long Korean narratives while minimizing data contamination. Built following MuSR, Ko-MuSR features fully Korean narratives, reasoning chains, and multiple-choice questions verified by human annotators for logical consistency and answerability. Evaluations of four large language models -- two multilingual and two Korean-specialized -- show that multilingual models outperform Korean-focused ones even in Korean reasoning tasks, indicating cross-lingual generalization of reasoning ability. Carefully designed prompting strategies, which combine few-shot examples, reasoning traces, and task-specific hints, further boost accuracy, approaching human-level performance. Ko-MuSR offers a solid foundation for advancing Korean NLP by enabling systematic evaluation of long-context reasoning and prompting strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。