用检索增强生成提升大模型拒答攻击能力，不需训练
Leveraging RAG for Training-Free Alignment of LLMs
- 基于检索增强生成，在推理时对比偏好样本进行对齐
- 在5个主流大模型上使代理攻击拒答率提升3.7倍
- 无需训练、兼容现有工具，适合安全防护场景
大型语言模型（LLM）对齐算法通常依赖偏好对的后训练。尽管这类方法广泛用于实现安全防护和符合通用人类偏好，我们发现当前最先进的对齐算法需要大量计算资源，且对近期代理攻击的拒答能力不足。为在不显著增加计算开销的前提下提升拒答能力，我们提出检索增强生成偏好对齐（RAG-Pref），一种基于RAG的简单对齐算法，通过在推理时引入偏好与非偏好样本的对比信息。RAG-Pref为在线（无需训练）方法，兼容现成工具包；当与离线（训练型）对齐算法结合使用时，在五个广泛使用的LLM上，相比其他在线对齐方法（2.9倍）和仅离线对齐（1.5倍），平均拒答率提升超过3.7倍。我们进一步表明，与其它在线对齐方法不同，RAG-Pref在通用人类偏好对齐任务上同样有效，且整体计算开销无显著增加。
原文摘要 · Abstract (English)
Large language model (LLM) alignment algorithms typically consist of post-training over preference pairs. While such algorithms are widely used to enable safety guardrails and align LLMs with general human preferences, we show that state-of-the-art alignment algorithms require significant computational resources while being far less capable of enabling refusal guardrails for recent agentic attacks. Thus, to improve refusal guardrails against such attacks without drastically increasing computational overhead, we introduce Retrieval Augmented Generation for Pref erence alignment (RAG-Pref), a simple RAG-based alignment algorithm which conditions on preferred and dispreferred samples to leverage contrastive information during inference. RAG-Pref is online (training-free), compatible with off-the-shelf packages, and, when combined with offline (training-based) alignment algorithms, enables more than an average 3.7 factor improvement in agentic attack refusals across five widely used LLMs, compared to 2.9 for other online alignment algorithms and 1.5 for offline alignment alone. We conclude by showing that, in stark contrast to other online alignment methods, RAG-Pref similarly increases performance on general human-preference alignment tasks and does not drastically increase overall computational requirements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。