arXiv:2510.04560cs.AI2025-10被引 12

让AI自动选对图文示例,提升多模态学习的准确性和适应性。

ContextNav: Towards Agentic Multimodal In-Context Learning

  • 用智能代理动态筛选图文示例,结合自动检索与人工筛选优势。
  • 在多个数据集上超越现有方法,尤其在噪声示例下表现更稳定。
  • 适合需要高效、可靠多模态任务适配的研究者与开发者。

近期研究显示,多模态大语言模型(MLLMs)具备强大的多模态上下文学习(ICL)能力,可从少量示例中快速适应新视觉-语言任务。然而,现有ICL方法在跨任务泛化与噪声鲁棒性之间难以平衡:手动选例虽精准但耗时且任务依赖,相似性检索虽可扩展却可能引入无关或结构不一致样本,降低性能。为此,我们提出ContextNav,首个将自动化检索与类人精炼结合的智能体框架,实现噪声鲁棒且动态优化的多模态ICL上下文构建。ContextNav通过图驱动的闭环流程,统一上下文管理与抗噪构造,建立资源感知的多模态嵌入管道,维护可检索向量库,并采用智能体检索与结构对齐生成抗干扰上下文。操作语法图(OGG)支持自适应工作流规划与优化,使智能体能根据下游ICL反馈调整策略。实验表明,ContextNav在多个数据集上达到当前最优性能,验证了智能体工作流在可扩展、鲁棒上下文构建中的潜力。

原文摘要 · Abstract (English)

Recent advances demonstrate that multimodal large language models (MLLMs) exhibit strong multimodal in-context learning (ICL) capabilities, enabling them to adapt to novel vision-language tasks from a few contextual examples. However, existing ICL approaches face challenges in reconciling scalability with robustness across diverse tasks and noisy contextual examples: manually selecting examples produces clean contexts but is labor-intensive and task-specific, while similarity-based retrieval improves scalability but could introduce irrelevant or structurally inconsistent samples that degrade ICL performance. To address these limitations, we propose ContextNav, the first agentic framework that integrates the scalability of automated retrieval with the quality and adaptiveness of human-like curation, enabling noise-robust and dynamically optimized contextualization for multimodal ICL. ContextNav unifies context management and noise-robust contextualization within a closed-loop workflow driven by graph-based orchestration. Specifically, it builds a resource-aware multimodal embedding pipeline, maintains a retrievable vector database, and applies agentic retrieval and structural alignment to construct noise-resilient contexts. An Operational Grammar Graph (OGG) further supports adaptive workflow planning and optimization, enabling the agent to refine its operational strategies based on downstream ICL feedback. Experimental results demonstrate that ContextNav achieves state-of-the-art performance across various datasets, underscoring the promise of agentic workflows for advancing scalable and robust contextualization in multimodal ICL.

多模态学习智能体上下文学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。