提出推理时安全对齐新方法，让大模型几乎必然生成安全回答。
On Almost Surely Safe Alignment of Large Language Models at Inference-Time
- 将安全生成建模为潜空间中的约束马尔可夫决策过程
- 通过动态惩罚机制实现接近必然的安全响应生成
- 无需修改模型权重，适合实际部署的轻量级安全对齐
我们提出一种新型的推理时对齐方法，旨在使大语言模型生成安全回应的概率趋近于1。该方法将安全生成建模为大语言模型潜空间中的约束马尔可夫决策过程（MDP），引入一个安全状态以追踪安全约束的演化，并动态惩罚不安全生成，确保生成结果的安全性。在潜空间中求解该MDP并施加足够大的惩罚后，可获得针对给定代价模型的形式化安全保证。基于此，我们提出了InferenceGuard，一种无需修改模型权重的实用实现方案。实验证明，InferenceGuard能有效平衡安全性与任务性能，在生成安全且对齐的回答方面优于现有推理时对齐方法。研究结果推动了大模型推理时安全对齐的发展，为资源密集、易过拟合的强化学习人类反馈（RLHF）等对齐技术提供了一种高效替代方案。
原文摘要 · Abstract (English)
We introduce a novel inference-time alignment approach for LLMs that aims to generate safe responses almost surely, i.e., with probability approaching one. Our approach models the generation of safe responses as a constrained Markov Decision Process (MDP) within the LLM's latent space. We augment a safety state that tracks the evolution of safety constraints and dynamically penalize unsafe generations to ensure the generation of safe responses. Consequently, we demonstrate formal safety guarantees w.r.t. the given cost model upon solving the MDP in the latent space with sufficiently large penalties. Building on this foundation, we propose InferenceGuard, a practical implementation that safely aligns LLMs without modifying the model weights. Empirically, we demonstrate that InferenceGuard effectively balances safety and task performance, outperforming existing inference-time alignment methods in generating safe and aligned responses. Our findings contribute to the advancement of safer LLM deployment through alignment at inference-time, thus presenting a promising alternative to resource-intensive, overfitting-prone alignment techniques like RLHF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。