arXiv:2602.04144cs.AIcs.LG2026-02被引 1

提出动态分步生成框架,解决多模态数据缺失时的失真问题。

OMG-Agent: Toward Robust Missing Modality Generation with Decoupled Coarse-to-Fine Agentic Workflows

  • 分三阶段处理:先规划语义,再检索证据,最后注入生成细节。
  • 在70%数据缺失下,比现有方法提升2.6分(CMU-MOSI)。
  • 适合高鲁棒性多模态系统开发,尤其应对极端数据缺失场景。

数据不完整严重损害多模态系统的可靠性。现有重建方法面临显著瓶颈:传统参数化/生成模型因过度依赖内部记忆易产生幻觉,而检索增强框架则受限于检索僵化。关键在于,这些端到端架构受制于语义-细节纠缠——逻辑推理与信号合成之间的结构性冲突,损害了生成保真度。本文提出全新框架OMG-Agent,将静态映射范式转变为动态粗粒度到细粒度的智能体工作流。通过模拟‘深思后行动’的认知过程,OMG-Agent明确将任务解耦为三个协同阶段:(1) 基于多模态大语言模型的语义规划器,通过渐进式上下文推理消除输入歧义,生成确定性结构化语义计划;(2) 非参数化证据检索器,将抽象语义锚定于外部知识;(3) 检索注入执行器,利用检索证据作为灵活特征提示,克服僵化并合成高保真细节。在多个基准上的大量实验表明,OMG-Agent持续优于当前最优方法,在极端缺失情况下仍保持鲁棒性,如在CMU-MOSI上70%缺失率下取得2.6点提升。

原文摘要 · Abstract (English)

Data incompleteness severely impedes the reliability of multimodal systems. Existing reconstruction methods face distinct bottlenecks: conventional parametric/generative models are prone to hallucinations due to over-reliance on internal memory, while retrieval-augmented frameworks struggle with retrieval rigidity. Critically, these end-to-end architectures are fundamentally constrained by Semantic-Detail Entanglement -- a structural conflict between logical reasoning and signal synthesis that compromises fidelity. In this paper, we present \textbf{\underline{O}}mni-\textbf{\underline{M}}odality \textbf{\underline{G}}eneration Agent (\textbf{OMG-Agent}), a novel framework that shifts the paradigm from static mapping to a dynamic coarse-to-fine Agentic Workflow. By mimicking a \textit{deliberate-then-act} cognitive process, OMG-Agent explicitly decouples the task into three synergistic stages: (1) an MLLM-driven Semantic Planner that resolves input ambiguity via Progressive Contextual Reasoning, creating a deterministic structured semantic plan; (2) a non-parametric Evidence Retriever that grounds abstract semantics in external knowledge; and (3) a Retrieval-Injected Executor that utilizes retrieved evidence as flexible feature prompts to overcome rigidity and synthesize high-fidelity details. Extensive experiments on multiple benchmarks demonstrate that OMG-Agent consistently surpasses state-of-the-art methods, maintaining robustness under extreme missingness, e.g., a $2.6$-point gain on CMU-MOSI at $70$\% missing rates.

多模态缺失数据生成框架智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。