XTC通过剔除高概率通用词提升生成多样性，效果显著且稳定。
XTC: Head-Aware Sampling by Excluding Top Choices
- 识别高置信度候选词，以概率ρ剔除最强选项，保留最弱合理选项
- 在4个模型上使多样性指标提升11%-38%，重复三元组减少27%-71%
- 适合追求创意与避免重复的生成任务，兼容主流推理框架
自回归语言模型的标准解码策略通过重缩放全量词表或截断低概率尾部来促进多样性，但忽略了开放生成中多个延续都合理却仍集中于最通用选择的常见情形。本文提出XTC（Exclude Top Choices），一种轻量级头感知解码算子，直接针对此场景。XTC设定绝对合理性阈值τ，当至少两个词满足条件时，以概率ρ移除占优的可选词，仅保留最弱合理选项再归一化。在Gemma 3 27B Q4、Gemma 3 12B Q6、DeepSeek R1 14B Q6及Llama 3.3 70B Q4共60项实验中，XTC显著改善了多样性-重复权衡。在创造性生成任务中，Distinct-2提升11–15%，重复三元组下降27–47%；结合温度调节后，Distinct-2增益达38%，重复三元组减少71%。盲测亚马逊众包研究显示，150名高级评分者中62.3%偏好XTC（p<10^-4），且流畅性未下降；GPT-4o评估结果与Anthropic人工判断一致。IFEval测试中，XTC在保持提示级严格准确率仅下降1.7个百分点的前提下，恢复大部分多样性收益；匹配Distinct-2的温度设置会使IFEval下降8.8分。该方法与温度、重复惩罚协同有效，对量化级别和模型族均具鲁棒性，跨十二类提示风格表现一致。XTC已被llama.cpp、ExLlamaV2及text-generation-webui采纳。
原文摘要 · Abstract (English)
Standard decoding rules for autoregressive language models promote diversity by rescaling the full next-token distribution or truncating its low-probability tail. These strategies overlook a common regime of open-ended generation in which several continuations are plausible but too much probability mass remains concentrated on the most generic choice. We introduce XTC (Exclude Top Choices), a lightweight head-aware decoding operator that targets this regime directly. XTC identifies tokens whose probabilities exceed an absolute plausibility threshold $τ$: when at least two qualify, it removes the dominant eligible choices with probability $ρ$ and retains only the weakest plausible alternative before renormalization. Across 60 experiments on Gemma 3 27B Q4, Gemma 3 12B Q6, and DeepSeek R1 14B Q6, with scaling validation on Llama 3.3 70B Q4, XTC improves the diversity-repetition Pareto frontier. On creative generation, Distinct-2 increases by 11--15% and repeat trigrams decrease by 27--47% across the four models. Combined with temperature scaling, gains reach 38% in Distinct-2 and 71% in repeat-trigram reduction over baseline. A blinded Amazon Mechanical Turk study with 150 Master raters yields a 62.3% creativity preference for XTC ($p<10^{-4}$) without reduced fluency, while a GPT-4o control judge reproduces the Anthropic-judge direction on every measure. On IFEval with Llama 3.3 70B Q4, XTC preserves prompt-level strict accuracy within 1.7 percentage points of baseline while recovering most of the diversity gain; a temperature setting matched on Distinct-2 reduces IFEval by 8.8 points. The effect is additive with temperature and repetition penalties, robust across quantization levels and model families, and consistent across twelve prompt genres. XTC has been adopted by llama.cpp, ExLlamaV2, and text-generation-webui.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。