通过模拟攻击与反馈机制，提升多轮视觉语言模型的安全性。
SaFeR-Steer: Evolving Multi-Turn MLLMs via Synthetic Bootstrapping and Feedback Dynamics
- 用合成数据逐步训练模型，模拟真实攻击演化过程。
- 多轮安全评测得分提升超30点，失败延迟至后期出现。
- 适合关注多轮对话安全的开发者与研究者。
多轮视觉语言模型（MLLMs）在实际应用中面临安全威胁，攻击者可通过不断演变的图文历史放大不安全意图，并利用长上下文导致的安全衰减。然而现有对齐方法仍依赖单轮数据和固定模板对话，造成训练与部署不匹配。为此，我们提出SaFeR-Steer，一种渐进式多轮对齐框架，结合分阶段合成数据生成与导师在环的GRPO算法，在动态策略下训练单一学生模型。引入轨迹一致的总结奖励（TCSR），聚合每轮最低与平均奖励，使低质量对话影响整体回报。我们发布新数据集STEER，包含12,934条SFT、2,000条RL及3,227条评测对话，涵盖2–10轮场景。实验基于Qwen2.5-VL-3B/7B，结果表明：单轮安全/助人度从48.30/45.86→81.84/70.77（3B），56.21/60.32→87.89/77.40（7B）；多轮对应指标从12.55/27.13→55.58/70.27（3B），24.66/46.48→64.89/72.35（7B），显著提升且失败延迟，实现超越规模的鲁棒性。代码已开源。
原文摘要 · Abstract (English)
MLLMs are increasingly deployed in multi-turn settings, where attackers can escalate unsafe intent through the evolving visual-text history and exploit long-context safety decay. Yet safety alignment is still dominated by single-turn data and fixed-template dialogues, leaving a mismatch between training and deployment. To bridge this gap, we propose SaFeR-Steer, a progressive multi-turn alignment framework that combines staged synthetic bootstrapping with tutor-in-the-loop GRPO to train a single student under adaptive, on-policy attacks. We also introduce Trajectory-Consistent Summative Reward (TCSR), which aggregates the historical minimum and average of turn rewards so that any low-quality turn affects the trajectory-level return. I. Dataset. We release STEER, a multi-turn multimodal safety dataset with STEER-SFT (12,934), STEER-RL (2,000), and STEER-Bench (3,227) dialogues spanning 2-10 turns. II. Experiment. Starting from Qwen2.5-VL-3B/7B, SaFeR-Steer substantially improves Safety/Helpfulness on both single-turn (48.30/45.86 $\rightarrow$ 81.84/70.77 for 3B; 56.21/60.32 $\rightarrow$ 87.89/77.40 for 7B) and multi-turn benchmarks (12.55/27.13 $\rightarrow$ 55.58/70.27 for 3B; 24.66/46.48 $\rightarrow$ 64.89/72.35 for 7B), shifting failures to later turns and yielding robustness beyond scaling alone. Code is available at https://anonymous.4open.science/r/SaFeR-Steer
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。