防御视觉语言动作机器人被物理注意力劫持,提升真实场景鲁棒性。
Structure-Aware Robust Fine-Tuning: Defending Vision-Language-Action Robots Against Physical Attention Hijacking

- 通过特征锚定与注意力修正,仅微调视觉编码器实现零推理开销防御。
- 在LIBERO上将攻击失败率从100%降至28.6%平均,真实机械臂成功率升至65%。
- 针对注意力劫持机制设计,适合关注机器人安全与对抗性攻击的研究者。
视觉-语言-动作(VLA)策略虽具通用操作潜力,但对现实世界攻击仍脆弱。我们发现,可物理实现的对抗补丁能通过触发‘关键动作-视觉注意力劫持’机制,使动作条件注意力偏离任务相关区域而聚焦于局部补丁。为此提出注意力引导语义破坏(AGSD)补丁,经期望-变换优化,可同时集中注意力并破坏视觉-语言语义对齐,具备强跨任务、跨架构迁移能力。为应对该威胁,提出结构感知鲁棒微调(SARF),仅微调视觉编码器,结合特征锚定、关键注意力修正及语言引导几何一致性约束,且仅作用于语义相关区域。在LIBERO数据集上,SARF将OpenVLA在AGSD攻击下的失败率从100%降至14.2%-56.8%(平均28.6%),同时保持原始性能;在真实PiPER机械臂上,成功率从23.0%提升至65.0%。结果表明,机制级鲁棒性是保障VLA机器人安全的有效路径。
原文摘要 · Abstract (English)
Vision-Language-Action (VLA) policies promise general robotic manipulation, but their robustness against physical-world attacks remains fragile. In particular, we show that physically realizable adversarial patches can reliably induce failures by triggering a mechanism we call policy-critical action-to-vision attention hijacking, where action-conditioned attention is diverted from task-relevant regions to a localized patch. To demonstrate the threat, we propose Attention-Guided Semantic Disruption (AGSD), an Expectation-over-Transformation (EOT) optimized printable patch that jointly (i) concentrates action-to-vision attention on the patch and (ii) disrupts vision-language semantic alignment, yielding strong cross-task and cross-architecture transfer. To mitigate such attacks, we introduce Structure-Aware Robust Fine-Tuning (SARF), a zero-inference-overhead defense that fine-tunes only the visual encoder using feature anchoring, policy-critical attention correction, and language-guided geometric consistency restricted to semantically relevant regions. On LIBERO, SARF reduces OpenVLA's failure rate under AGSD from 100% to 14.2%-56.8% (28.6% average) across suites while preserving clean performance, and on a real PiPER manipulator it improves average success under AGSD from 23.0% to 65.0%. These results highlight mechanism-level robustness as a practical path to securing VLA robots against physical attention hijacking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。