用精简表示替代图片,提升多模态搜索推荐的效率与准确率
FUSE : Failure-aware Usage of Subagent Evidence for MultiModal Search and Recommendation
- 用结构化数据替代图像输入,降低计算开销
- 上下文压缩策略在各项指标上均表现最佳,召回率达99.4%
- 适合需要高效多模态交互的智能创作系统开发
多模态创意助手将用户目标分解并路由至子代理完成布局、风格、检索和生成。检索质量至关重要,但理解意图、选择内容类型、候选查找(召回)或排序均可能失败。同时,图像传输与处理成本高,使朴素多模态方法不可行。本文提出FUSE:一种面向多模态搜索与推荐的故障感知子代理证据利用方法。FUSE将大部分原始图像提示替换为紧凑的地面设计表示(GDR):由规划团队提供的包含元素类型(图像、文本、形状、图标、视频、标志)、结构、样式、显著颜色及用户选择的选区感知JSON。FUSE实现七种上下文预算策略:全面基线提示、上下文压缩、思维链推理、小样本优化、检索增强上下文、两阶段处理、零样本极简主义。最后,通过管道归因层将子代理信号转化为简单检查:意图对齐、内容类型/路由合理性、召回健康度(如零命中率与顶匹配强度)、排名位移分析。我们在788个来自不同用户和设计模板的查询上评估了七种上下文预算变体。系统评估显示,上下文压缩在所有阶段表现最优,意图准确率达93.3%,路由成功率(含回退)达86.8%,召回率达99.4%,NDCG@5为88.5%。结果表明,有策略的上下文摘要优于全面或极简的上下文策略。
原文摘要 · Abstract (English)
Multimodal creative assistants decompose user goals and route tasks to subagents for layout, styling, retrieval, and generation. Retrieval quality is pivotal, yet failures can arise at several stages: understanding user intent, choosing content types, finding candidates (recall), or ranking results. Meanwhile, sending and processing images is costly, making naive multimodal approaches impractical. We present FUSE: Failure-aware Usage of Subagent Evidence for MultiModal Search and Recommendation. FUSE replaces most raw-image prompting with a compact Grounded Design Representation (GDR): a selection aware JSON of canvas elements (image, text, shape, icon, video, logo), structure, styles, salient colors, and user selection provided by the Planner team. FUSE implements seven context budgeting strategies: comprehensive baseline prompting, context compression, chain-of-thought reasoning, mini-shot optimization, retrieval-augmented context, two-stage processing, and zero-shot minimalism. Finally, a pipeline attribution layer monitors system performance by converting subagent signals into simple checks: intent alignment, content-type/routing sanity, recall health (e.g., zero-hit and top-match strength), and ranking displacement analysis. We evaluate the seven context budgeting variants across 788 evaluation queries from diverse users and design templates (refer Figure 3). Our systematic evaluation reveals that Context Compression achieves optimal performance across all pipeline stages, with 93.3% intent accuracy, 86.8% routing success(with fallbacks), 99.4% recall, and 88.5% NDCG@5. This approach demonstrates that strategic context summarization outperforms both comprehensive and minimal contextualization strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。