只在关键节点干预,让大模型生成更准且更快
Inference-time Alignment via Sparse Junction Steering
- 仅在高熵决策点稀疏干预,避免全程修改
- 干预20%~80%的词元即可达到顶尖对齐效果
- 适合追求高效对齐与高质量生成的开发者
词元级引导已成为推理时对齐的关键方法,可在不更新参数的情况下精细调控大语言模型的输出分布。现有方法依赖每一步解码都进行密集干预,不仅计算开销大,还可能因过度偏离模型固有分布而降低生成质量。本文提出稀疏推理时对齐(SIA),仅在生成轨迹中的关键决策点进行干预。核心洞察是:高熵节点是生成路径中易产生偏差的关键点,需在此类位置引入对齐奖励信号。跨多种模型和对齐目标的实验表明,仅干预20%至80%的词元即可实现更优的对齐-效率权衡。对于Qwen3等强基座模型,仅干预20%词元的效果就可媲美甚至超越经过重度微调的指令模型。该稀疏策略既能增强引导能力,又能更好保持模型原生分布,兼容Best-of-N等搜索方法,计算成本最高降低6倍。
原文摘要 · Abstract (English)
Token-level steering has emerged as a pivotal approach for inference-time alignment, enabling fine grained control over large language models by modulating their output distributions without parameter updates. While effective, existing methods rely on dense intervention at every decoding step. This persistent manipulation not only incurs substantial computational overhead but also risks compromising generation quality by excessively drifting from the model's intrinsic distribution. In this work, we show that dense intervention is unnecessary and propose Sparse Inference time Alignment (SIA), which performs sparse junction steering by intervening only at critical decision points along the generation trajectory. Our key insight is that high entropy junctions mark pivotal decision points in the generation trajectory and are particularly susceptible to misalignment, indicating the need to introduce alignment related reward signals at these points. Extensive experiments across different model families and alignment objectives show that steering only 20% to 80% of tokens achieves superior alignment-efficiency trade offs. For strong base models such as Qwen3, intervening on as few as 20% of tokens matches or even surpasses heavily post-trained instruct models. This sparsity enables stronger guidance while better preserving the model's native distribution, integrates seamlessly with search based methods such as Best-of-N, and reduces computational cost by up to 6x.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。