arXiv:2605.05415cs.LGcs.AI2026-05

用信息论方法动态强化难样本,提升大模型抗提示攻击能力。

Information Theoretic Adversarial Training of Large Language Models

论文配图:Information Theoretic Adversarial Training of Large Language Models
图 1 · 摘自论文原文
  • 基于分布鲁棒优化,通过f-散度自动加权难样本。
  • 在多个模型上将攻击成功率显著降低,计算开销与现有方法相当。
  • 适合需要高效鲁棒对齐的大模型训练场景。

大语言模型尽管在对齐与安全方面取得进展,仍易受新型对抗性提示攻击影响,表现出有害行为。现有对抗训练方法计算成本高且难以扩展。近期连续对抗训练方法(如CAT、CAPO)利用嵌入空间的梯度扰动,提升了效率与表达力。本文提出WARDEN,一种分布鲁棒的对抗训练框架,通过围绕经验数据分布的f-散度模糊集动态重加权对抗样本。该方法在分布球内优化最坏情况下的对抗损失,自动聚焦更难样本。借助凸对偶形式,目标函数转化为KL散度下的log-sum-exp形式,动态参数控制重加权强度。本研究催生一类新的信息论目标函数,在多个大模型和攻击设置下显著降低攻击成功率,同时保持模型效用,计算与实用成本与CAT、CAPO、MixAT相当,是一种可扩展的鲁棒对齐实用方案。

原文摘要 · Abstract (English)

Large language models (LLMs) remain vulnerable to adversarial prompting despite advances in alignment and safety, often exhibiting harmful behaviors under novel attack strategies. While adversarial training can improve robustness, existing approaches are computationally expensive and difficult to scale. Recent continuous adversarial training methods, such as Continuous adversarial training (CAT) and Continuous Adversarial Preference Optimization (CAPO), address this challenge by leveraging gradient-based perturbations in the embedding space, enabling more efficient and expressive attacks. Building on this paradigm, we propose WARDEN, a distributionally robust adversarial training framework for LLMs that dynamically reweights adversarial examples through an f -divergence ambiguity set around the empirical training distribution. Our method optimizes the worst-case adversarial loss within a divergence ball around the empirical data distribution, automatically emphasizing harder adversarial examples. Using the convex dual formulation, the objective reduces to a log-sum-exp form under the KL divergence, with a dynamical parameter controlling the strength of reweighting. This study leads to a new class of information-theoretic objectives that significantly reduce attack success rates while maintaining model utility. Across multiple LLMs and attack settings, WARDEN substantially reduces attack success rates with computational and utility costs comparable to CAT-, CAPO-, and MixAT-based baselines, making it a practical approach for scalable robust alignment.

大模型安全对抗训练信息论鲁棒对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。