让大模型真正理解输出长度,精准执行长度指令。
LARFT: Closing the Cognition-Action Gap for Length Instruction Following in Large Language Models
- 用事后自知机制训练模型识别自身生成长度。
- 在3个长度指令任务上平均提升20.92分,通用能力仅降1.45分。
- 适合需要严格控制输出长度的对话、摘要等场景。
尽管大语言模型在复杂指令遵循任务中表现优异,但精确控制输出长度仍是一个持续挑战。现有方法主要通过外部施加长度信号或优化目标来强制约束,却忽视了模型内在的长度认知缺陷。为此,我们提出LARFT(长度感知强化微调)训练框架,将模型的长度认知与行为对齐。具体而言,LARFT结合面向长度的强化学习与事后长度自知机制,将在线数据转化为事后自我意识任务,使模型学习识别自身生成的实际长度。该方法联合优化模型内部的长度表征与策略,从而实现精确可靠的长度指令遵循。在四个基础模型上的大量实验表明,LARFT优于现有基线,在三个长度指令遵循基准上平均提升20.92分,仅在四个通用能力基准上略有下降(-1.45分)。
原文摘要 · Abstract (English)
Despite the strong performance of Large Language Models (LLMs) on complex instruction-following tasks, precise control of output length remains a persistent challenge. Existing methods primarily attempt to enforce length constraints by externally imposing length signals or optimization objectives, while largely overlooking the underlying limitation: the model's intrinsic deficit in length cognition. To address this, we propose LARFT (Length-Aware Reinforcement Fine-Tuning), a training framework that aligns the model's length cognition with its action. Specifically, LARFT integrates length-oriented reinforcement learning with a hindsight length awareness. By transforming on-policy data into hindsight self-awareness tasks where the model learns to identify the actual length of its own generation, LARFT jointly optimizes the model's internal representation of length information and refines its policy to satisfy length constraints, thereby achieving precise and reliable length instruction following. Extensive experiments across four base models demonstrate that LARFT outperforms existing baselines, achieving an average improvement of +20.92 points across three length instruction following benchmarks with only a marginal decline of -1.45 points on four general capability benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。