arXiv:2606.11201cs.LGcs.AI2026-06ACL被引 1

通过概率融合提升大模型推理时对齐效果,减少无效干预

To Intervene or Not: Guiding Inference-time Alignment with Probabilistic Model Blending

论文配图:To Intervene or Not: Guiding Inference-time Alignment with Probabilistic Model Blending
图 1 · 摘自论文原文
  • 用混合分布代替二元判断,动态加权可靠模型贡献
  • 在困难模型对上性能提升最高达50%且避免过度干预
  • 适合需要稳定对齐的落地场景,尤其对弱对齐模型有效

大语言模型的大规模部署使得推理时对齐成为必要,以确保新训练模型能安全有效地响应用户指令。现有方法通常在生成阶段仅依赖特定对齐模型提供的指导,但未评估其可靠性。我们系统性评估发现,不同模型的指导效果差异显著;无效指导会导致更多混乱,进而引发更多干预,反而降低性能。为提升干预效率,我们提出BlendIn框架,将二元决策转向构建融合两模型知识的混合分布。该框架基于可靠性进行质量感知对齐,按可信度比例加权各模型贡献。相比现有方法,它保留有益指导并抑制不可靠建议。BlendIn提供误对齐指导的诊断信号与缓解策略,在挑战性模型对上实现一致性能提升,最高达50%。代码已开源:https://github.com/DecayingSeart/BlendIn。

原文摘要 · Abstract (English)

The wide deployment of LLMs has made model alignment necessary to make newly trained models safely and effectively respond to user instructions. Among different methods, inference-time alignment is often cheaper as it intervenes (i.e., offers guidances) only during output generation. Existing proposals apply guidances extracted from certain aligned models without properly assessing their reliability. Nonetheless, our systematic evaluation reveals that guidance effectiveness varies drastically across models; since ineffective guidances lead to further confusion and thus further interventions, the resulting excessive interventions typically indicate poor performance. To make interventions more effective and thus more efficient, we introduce BlendIn, an inference-time alignment framework that shifts from binary decisions to creating hybrid distributions integrating both models' knowledge. BlendIn stabilizes inference-time alignment by performing quality-aware alignment and proportionally weighting each model's contribution based on reliability. Compared with existing works, it preserves beneficial guidance while downweighting unreliable suggestions. BlendIn provides both diagnostic signals and mitigation strategies for misaligned guidance, achieving consistent and up to 50% performance improvement on challenging model pairs. Our code is available at: https://github.com/DecayingSeart/BlendIn.

模型对齐推理优化概率融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。