arXiv:2505.16234cs.CLcs.AI2025-05NeurIPS被引 10

测试大模型写长文本时能否听懂字数要求,发现多数模型做不到。

LIFEBench: Evaluating Length Instruction Following in Large Language Models

  • 构建多语言、多任务的长文本指令评测集,覆盖16到8192词长度
  • 26个主流模型在超长输出时普遍失败,连宣称支持32K的也达不到
  • 推理型模型反而比专用长文本生成模型更擅长遵守字数要求

尽管大语言模型(LLMs)能处理长上下文的博士级推理问题,但在执行显式长度指令——如撰写一篇10,000字小说——时仍表现不佳。现有基准主要评估生成质量,却忽视输出是否满足长度约束。为此,我们提出长度指令遵循评测基准LIFEBench,全面评估模型在多样化任务和广泛长度范围内的长度指令遵循能力。LIFEBench包含4类任务、中英文共10,800个实例,覆盖16至8192词的长度约束。我们评估了26个广泛使用的LLM,发现大多数模型仅能合理响应短长度指令,超出一定阈值后性能急剧下降。令人惊讶的是,几乎所有模型在实际中都无法达到厂商声称的最大输出长度,我们的扩展评估甚至验证到32,000词仍无法实现。即使具备长上下文能力的LLM,也未能有效提升长度指令遵循能力。值得注意的是,推理型模型的表现优于专门的长文本生成模型,达到当前最优水平。总体而言,LIFEBench揭示了当前LLMs在长度指令遵循方面的根本性局限,为未来研究提供了关键洞见。

原文摘要 · Abstract (English)

While large language models (LLMs) can solve PhD-level reasoning problems over long context inputs, they still struggle with a seemingly simpler task: following explicit length instructions-e.g., write a 10,000-word novel. Additionally, models often generate far too short outputs, terminate prematurely, or even refuse the request. Existing benchmarks focus primarily on evaluating generations quality, but often overlook whether the generations meet length constraints. To this end, we introduce Length Instruction Following Evaluation Benchmark (LIFEBench) to comprehensively evaluate LLMs' ability to follow length instructions across diverse tasks and a wide range of specified lengths. LIFEBench consists of 10,800 instances across 4 task categories in both English and Chinese, covering length constraints ranging from 16 to 8192 words. We evaluate 26 widely-used LLMs and find that most models reasonably follow short-length instructions but deteriorate sharply beyond a certain threshold. Surprisingly, almost all models fail to reach the vendor-claimed maximum output lengths in practice, as further confirmed by our evaluations extending up to 32K words. Even long-context LLMs, despite their extended input-output windows, counterintuitively fail to improve length-instructions following. Notably, Reasoning LLMs outperform even specialized long-text generation models, achieving state-of-the-art length following. Overall, LIFEBench uncovers fundamental limitations in current LLMs' length instructions following ability, offering critical insights for future progress.

大模型评测长度控制指令遵循

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。