arXiv:2608.22761cs.CLcs.AI2026-08

提出新方法抑制大模型生成时重复已有内容,提升文本多样性。

Don't Repeat Yourself: Stopping Verbatim Loops at Sampling Time

  • 在采样时通过调整概率惩罚重复上下文的完整片段
  • 使循环率降低47%,同时保持模型原有性能
  • 适合需要流畅长文本生成的研究与应用

大型语言模型在自回归生成中易出现重复已存在内容的循环现象。传统防重策略基于词元重复,难以区分结构循环,常损害文本格式与流畅性。本文提出采样阶段的逻辑修正方法 DRY,仅当候选词会延续当前后缀为先前上下文中的精确重复时才施加惩罚。该方法可有效保护对话模板和格式标记。在1.5B至120B参数模型、九类提示任务及600对人类评估中,DRY将后缀扩展率降低47%,并提升词汇多样性。对照实验表明,机制核心在于后缀匹配。在AWQ量化后的70B和120B模型上,循环率减少约一半,且在MT-Bench、MMLU和GSM8K测试中性能无损。该方法已被llama.cpp、ExLlamaV2和text-generation-webui等主流开源推理框架采纳。

原文摘要 · Abstract (English)

Large Language Models generate text autoregressively, but open-ended generation is prone to verbatim looping, in which models repeat spans already present in context. Standard defenses such as repetition, presence, and frequency penalties and n-gram blocking act on token recurrence rather than the sequential structure of a loop, and often suppress looping only at strengths that also degrade formatting or fluency. We propose Don't Repeat Yourself (DRY), a sampling-time logit adjustment that penalizes a candidate token only when generating it would extend the current suffix into an exact continuation of a span seen earlier in the context. Sequence breakers protect chat templates and formatting tokens. Across models from 1.5B to 120B parameters, nine prompt families, and a 600-pair human study, DRY reduces suffix-extension rate by 47% while improving lexical diversity. An intervention-matched placebo produces no comparable reduction, identifying suffix matching as the operative mechanism. On AWQ-quantized 70B and 120B models, DRY reduces loop rate by roughly half while preserving MT-Bench, MMLU, and GSM8K performance, whereas standard alternatives lose measurable ground. DRY has been adopted by popular open-source LLM inference frameworks including llama.cpp, ExLlamaV2, and text-generation-webui, highlighting its practical impact on text generation.

语言模型文本生成防重复采样优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。