对比推理模型在机器翻译与摘要评估中的表现,发现效果因模型和任务而异。
DeepSeek-R1 vs. o3-mini: How Well can Reasoning LLMs Evaluate MT and Summarization?
- 用推理与非推理模型对比评估生成文本质量,涵盖多种参数规模。
- o3-mini 在翻译评估中随推理增强表现更好,DeepSeek-R1 仅在摘要一致性上胜出。
- 模型越小越易退化,且推理消耗与评分质量并非总是正相关。
具备推理能力的大语言模型在逻辑任务中表现优异,但其在自然语言生成评估中的作用尚未明确。本研究系统比较了推理型与非推理型模型在机器翻译与文本摘要评估任务中的表现。我们评估了八种模型,包括最先进的推理模型(DeepSeek-R1、OpenAI o3)、其压缩版本(8B-70B 参数)以及对应的非推理模型。在 WMT23 与 SummEval 基准测试中发现:模型架构与任务类型决定收益——o3-mini 模型在翻译评估中随推理增强表现提升,而 DeepSeek-R1 除摘要一致性评估外普遍弱于其非推理版本。相关性分析显示,推理令牌使用量仅在特定模型中与评估质量相关;几乎所有模型在识别更多质量问题时都会增加推理消耗。模型蒸馏在 32B 规模下仍保持合理性能,但在 8B 规模下显著下降。本工作首次系统评估推理模型在 NLG 评估中的表现,并提供代码以推动后续研究:https://github.com/NL2G/reasoning-eval。
原文摘要 · Abstract (English)
Reasoning-enabled large language models (LLMs) excel in logical tasks, yet their utility for evaluating natural language generation remains unexplored. This study systematically compares reasoning LLMs with non-reasoning counterparts across machine translation and text summarization evaluation tasks. We evaluate eight models spanning state-of-the-art reasoning models (DeepSeek-R1, OpenAI o3), their distilled variants (8B-70B parameters), and equivalent non-reasoning LLMs. Experiments on WMT23 and SummEval benchmarks reveal architecture and task-dependent benefits: OpenAI o3-mini models show improved performance with increased reasoning on MT, while DeepSeek-R1 and generally underperforms compared to its non-reasoning variant except in summarization consistency evaluation. Correlation analysis demonstrates that reasoning token usage correlates with evaluation quality only in specific models, while almost all models generally allocate more reasoning tokens when identifying more quality issues. Distillation maintains reasonable performance up to 32B parameter models but degrades substantially at 8B scale. This work provides the first assessment of reasoning LLMs for NLG evaluation and comparison to non-reasoning models. We share our code to facilitate further research: https://github.com/NL2G/reasoning-eval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。