让大模型拒绝越狱攻击，关键在锁定恶意意图。
From Shallow to Deep: Pinning Semantic Intent via Causal GRPO
- 用因果探针分离意图与表达风格，识别隐藏恶意信号。
- 通过累计因果惩罚，使模型学会有害词积累即降分。
- 既抗越狱又不损通用能力，适合安全强化场景。
大语言模型虽在标准评测中表现安全，但仍易受前缀攻击（如“当然，以下是”）影响。我们诊断此问题源于‘浅层安全对齐’，其根源为语义表征衰减：模型生成合规前缀时，内部恶意意图信号逐渐消失。为此，提出两阶段因果GRPO（TSC-GRPO）框架。首先基于因果可识别性理论，训练因果意图探针以分离不变意图与风格扰动；其次通过群体相对策略优化，将该因果认知内化至策略中。在‘岔路’训练场景中引入累积因果惩罚，强制模型学习到有害词积累会单调降低奖励，从而实现鲁棒的后期拒绝。实验表明，TSC-GRPO显著优于基线，在抵御越狱攻击的同时保持通用能力。
原文摘要 · Abstract (English)
Large Language Models remain vulnerable to adversarial prefix attacks (e.g., ``Sure, here is'') despite robust standard safety. We diagnose this vulnerability as Shallow Safety Alignment, stemming from a pathology we term semantic representation decay: as the model generates compliant prefixes, its internal malicious intent signal fades. To address this, we propose Two-Stage Causal-GRPO (TSC-GRPO), a framework designed to achieve intent pinning. First, grounded in causal identifiability theory, we train a causal intent probe to disentangle invariant intent from stylistic perturbations. Second, we internalize this causal awareness into the policy via Group Relative Policy Optimization. By employing a cumulative causal penalty within ``fork-in-the-road'' training scenarios, we force the model to learn that accumulating harmful tokens monotonically decreases reward, enabling robust late-stage refusals. Experiments show that TSC-GRPO significantly outperforms baselines in defending against jailbreak attacks while preserving general utility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。