提出新方法提升开源大模型抗恶意前缀攻击能力
Matching Ranks Over Probability Yields Truly Deep Safety Alignment
- 通过匹配词元排名而非概率，增强安全对齐
- 在三种主流模型上使安全防护提升4.7倍
- 适合关注开源模型安全性的研究者与开发者
开源大语言模型在推动AI民主化中至关重要，但其开放性也易被恶意用户滥用。已有研究显示，仅在生成前添加肯定前缀即可绕过前沿开源模型的安全对齐。近期一种基于监督微调的数据增强防御方法可实现‘深层’安全对齐，使模型在接收到有害前缀后立即生成自然语言拒绝。本文揭示了一种更通用的前缀攻击——秩辅助前缀攻击（RAP），通过在每步选取预测前20个词元中低概率的有害词元（忽略高概率的拒绝词元）来提取有害内容。为此，我们提出一种新思路：匹配数据增强目标分布中的词元排名（而非概率），提出名为PRESTO的新方法，通过正则化注意力对有害前缀词元的关注程度来强化深层对齐。相比单纯使用数据增强，PRESTO在三种主流模型上对RAP攻击的安全性提升达4.7倍。该工作为构建更安全的开源模型提供了可行路径。
原文摘要 · Abstract (English)
Open-source Large Language Models (LLMs) play a critical role in the democratization of AI, yet their "open" nature introduces more avenues for malicious actors to misuse them for harmful purposes. A frustratingly easy but powerful technique known as the prefilling attack has been shown to effectively circumvent the safety alignment of frontier open-source LLMs by simply prefilling the assistant response with an affirmative prefix before decoding. A recent promising supervised fine-tuning defense proposes using a simple data augmentation scheme to achieve a "deep" safety alignment, allowing the model to generate natural language refusals immediately following harmful prefills. In this work, we show that a simple generalization of the prefilling attack, which we refer to as the Rank-Assisted Prefilling (RAP) attack, can effectively extract harmful content from models fine-tuned with the data augmentation defense by selecting low-probability "harmful" tokens from the top 20 predicted next tokens at each step (thus ignoring high-probability \enquote{refusal} tokens). We then propose a new perspective on achieving deep safety alignment by matching the token ranks in the underlying data augmentation target distribution (rather than just their probabilities), yielding a surprisingly simple approach to strengthening deep alignment we call PRefill attEntion STOpping (PRESTO) that regularizes the attention placed on harmful prefill tokens. Compared to just using data augmentation, PRESTO yields up to a 4.7x improvement in safety under RAP attacks across three popular LLMs. By achieving a stronger level of safety against practical and accessible attacks, our work paves a path towards safer open-source models. The source code accompanying this paper can be found at: https://github.com/uiuc-focal-lab/push-forward-alignment
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。