让大模型会搜索:用新方法解决长对话中记忆爆炸问题
POINTS-Seeker: An Open Recipe for Multimodal Search Agents with Visual Memory Management

- 通过训练让模型从被动理解变为主动搜索,学会规划和用工具
- 在六个基准测试中达到同规模模型最好水平,尤其擅长多轮复杂查询
- 适合做智能助手、自动调研或需要长期记忆的视觉搜索任务
大型多模态模型虽擅长视觉理解,但面对实时、知识密集型问题时受限于静态参数化知识。现有多模态搜索代理开发困难,主要因缺乏从零构建搜索能力的公开方案,以及长周期多轮交互导致的上下文膨胀(注意力稀释)。本文提出原生多模态搜索代理POINTS-Seeker-8B,首次提供可复现的训练流程:引入“代理播种”阶段,从非代理基础模型中直接培养工具使用与规划能力;为突破上下文瓶颈,提出V-Fold自适应记忆管理机制——将近期交互以高保真文本保留,旧历史信息通过渲染折叠进视觉空间,利用跨模态对齐保存原始证据,避免文本冗余。实验显示,该模型在六项多模态搜索基准上达到同规模模型最优表现。
原文摘要 · Abstract (English)
Large Multimodal Models (LMMs) excel at visual perception but struggle with real-time, knowledge-intensive queries due to their reliance on static parametric knowledge. While multimodal search agents offer a promising solution, developing them from vanilla LMMs presents two major challenges: the lack of open recipes to cultivate search agency from scratch, and the severe context explosion (attention dilution) that occurs during long-horizon multi-turn investigations. In this paper, we address both challenges by developing a native multimodal search agent. First, we introduce an open training recipe featuring Agentic Seeding, a formative training stage that bootstraps tool-use and planning abilities directly from a non-agentic foundation. Second, to overcome the context bottleneck, we propose V-Fold, an adaptive memory management mechanism. V-Fold retains recent interactions as high-fidelity text while folding stale historical context into the visual space via rendering, exploiting the model's cross-modal alignment to preserve raw evidence without text token redundancy. Combining these innovations, we present POINTS-Seeker-8B, which achieves state-of-the-art performance among models of comparable scale across six multimodal search benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。