arXiv:2603.15800cs.CVcs.CL2026-03中稿 · CVPR被引 4

让多模态大模型通过自我反思记忆,识别隐含安全意图差异。

Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory

  • 构建自反记忆库,动态积累过往安全判断经验。
  • 在多个基准上显著提升上下文安全识别准确率。
  • 适合关注模型安全与推理鲁棒性的研究者使用。

多模态大语言模型在视觉推理任务中表现卓越,但其安全风险仍令人担忧。现有研究多聚焦于检测并拒绝明显不安全输入,却忽视了上下文安全——即模型需区分表面相似但安全意图迥异的场景。本文提出MM-SafetyBench++,一个精心设计的上下文安全评估基准:针对每组不安全图文对,通过最小修改生成对应的安全版本,仅反转用户意图而保留上下文语义,实现可控评估。进一步提出EchoSafe,一种无需训练的框架,通过维护自反记忆库,从历史交互中累积并检索安全洞察,将相关经验融入当前提示,实现推理时的上下文感知与安全行为持续演化。在多个多模态安全基准上的实验表明,EchoSafe性能显著优于基线,为推进多模态大模型的上下文安全提供了强基准。所有数据与代码已开源。

原文摘要 · Abstract (English)

Multi-modal Large Language Models (MLLMs) have achieved remarkable performance across a wide range of visual reasoning tasks, yet their vulnerability to safety risks remains a pressing concern. While prior research primarily focuses on jailbreak defenses that detect and refuse explicitly unsafe inputs, such approaches often overlook contextual safety, which requires models to distinguish subtle contextual differences between scenarios that may appear similar but diverge significantly in safety intent. In this work, we present MM-SafetyBench++, a carefully curated benchmark designed for contextual safety evaluation. Specifically, for each unsafe image-text pair, we construct a corresponding safe counterpart through minimal modifications that flip the user intent while preserving the underlying contextual meaning, enabling controlled evaluation of whether models can adapt their safety behaviors based on contextual understanding. Further, we introduce EchoSafe, a training-free framework that maintains a self-reflective memory bank to accumulate and retrieve safety insights from prior interactions. By integrating relevant past experiences into current prompts, EchoSafe enables context-aware reasoning and continual evolution of safety behavior during inference. Extensive experiments on various multi-modal safety benchmarks demonstrate that EchoSafe consistently achieves superior performance, establishing a strong baseline for advancing contextual safety in MLLMs. All benchmark data and code are available at https://echosafe-mllm.github.io.

多模态安全评估自反思推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。