arXiv:2510.26697cs.CLcs.AI2025-10被引 6

让大模型自己决定如何生成,告别手动调参。

The End of Manual Decoding: Towards Truly End-to-End Language Models

  • 用轻量头动态预测每步的温度与top-p值,实现自适应采样。
  • 在8个基准上超越默认策略,接近人工调优的最优表现。
  • 能理解'低随机性生成'等指令,实现交互式可控生成。

当前大模型所谓的‘端到端’其实依赖不可微的解码过程,需人工反复调整温度、top-p等超参数。本文提出AutoDeco,通过在标准Transformer中添加轻量级模块,每一步动态预测上下文相关的温度和top-p值,将解码变为可微的、逐标记的过程,实现单次前向传播内的自我调节。在8个基准测试中,AutoDeco显著优于默认解码策略,性能接近通过‘测试集作弊’获得的最优人工调参基线——这是静态方法的实际上限。更关键的是,模型展现出指令驱动解码的涌现能力:能理解自然语言指令(如‘以低随机性生成’),并据此逐标记调整采样策略,开启可调控、可交互的大模型生成新范式。

原文摘要 · Abstract (English)

The "end-to-end" label for LLMs is a misnomer. In practice, they depend on a non-differentiable decoding process that requires laborious, hand-tuning of hyperparameters like temperature and top-p. This paper introduces AutoDeco, a novel architecture that enables truly "end-to-end" generation by learning to control its own decoding strategy. We augment the standard transformer with lightweight heads that, at each step, dynamically predict context-specific temperature and top-p values alongside the next-token logits. This approach transforms decoding into a parametric, token-level process, allowing the model to self-regulate its sampling strategy within a single forward pass. Through extensive experiments on eight benchmarks, we demonstrate that AutoDeco not only significantly outperforms default decoding strategies but also achieves performance comparable to an oracle-tuned baseline derived from "hacking the test set"-a practical upper bound for any static method. Crucially, we uncover an emergent capability for instruction-based decoding control: the model learns to interpret natural language commands (e.g., "generate with low randomness") and adjusts its predicted temperature and top-p on a token-by-token basis, opening a new paradigm for steerable and interactive LLM decoding.

端到端生成自适应采样指令控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。