用强化学习生成通用对抗后缀，可跨任务模型有效干扰大模型输出。
Universal Adversarial Suffixes for Language Models Using Reinforcement Learning with Calibrated Reward
- 将后缀视为策略，用PPO算法在冻结模型上训练。
- 在5个数据集上使模型准确率下降，跨模型迁移能力更强。
- 通过校准奖励消除标签偏差，提升攻击泛化性，适合安全测试场景。
语言模型易受短对抗后缀影响而改变预测结果。以往方法多依赖梯度搜索或规则，但脆弱且局限于特定任务或模型。本文提出一种强化学习框架，将后缀视为策略，使用近端策略优化（PPO）在冻结模型作为奖励代理的情况下进行训练。采用校准交叉熵进行奖励塑造，消除标签偏差并聚合不同表面形式，增强迁移性。在涵盖情感分析、自然语言推理、改写和常识推理的五个主流NLP数据集上评估，使用Qwen2-1.5B Instruct、TinyLlama-1.1B Chat和Phi-1.5三种模型。结果表明，强化学习训练的后缀能持续降低准确率，并在任务与模型间表现出更强的迁移能力，优于同类现有触发器。
原文摘要 · Abstract (English)
Language models are vulnerable to short adversarial suffixes that can reliably alter predictions. Previous works usually find such suffixes with gradient search or rule-based methods, but these are brittle and often tied to a single task or model. In this paper, a reinforcement learning framework is used where the suffix is treated as a policy and trained with Proximal Policy Optimization against a frozen model as a reward oracle. Rewards are shaped using calibrated cross-entropy, removing label bias and aggregating across surface forms to improve transferability. The proposed method is evaluated on five diverse NLP benchmark datasets, covering sentiment, natural language inference, paraphrase, and commonsense reasoning, using three distinct language models: Qwen2-1.5B Instruct, TinyLlama-1.1B Chat, and Phi-1.5. Results show that RL-trained suffixes consistently degrade accuracy and transfer more effectively across tasks and models than previous adversarial triggers of similar genres.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。