arXiv:2607.25294cs.CVcs.AI2026-07被引 1

评测多模态上下文学习能力,揭示模型在图文结合任务中的短板。

CLBench-V: Evaluating Multimodal Context Learning from Grounding to Knowledge Acquisition

论文配图:CLBench-V: Evaluating Multimodal Context Learning from Grounding to Knowledge Acquisition
图 1 · 摘自论文原文
  • 构建跨领域的多模态上下文学习基准CLBench-V,涵盖科学、金融等六大场景。
  • 3443个实例测试显示最优模型准确率仅0.2847,表明该能力仍远未成熟。
  • 发现不同模型在上下文定位、新信息应用、知识获取上各有优劣。

现实任务常需模型从特定上下文中学习,而非仅依赖预训练知识。尽管近期研究关注上下文学习,但现有评估多集中于文本上下文。实际中,上下文常为多模态:科学结论通过图表传达,财务指标分散于转换报告,空间决策依赖地图、场景或网页。为此,我们提出CLBench-V,一个面向多模态上下文学习的基准,围绕上下文定位、新信息应用和新知识学习三个维度设计任务。该基准融合公开数据集与新构建数据集,覆盖科学、金融、长文档理解、空间推理及基于网页的视觉问答等领域。为降低构建成本,采用自动化生成与筛选流程。在3,443个实例和六种先进多模态模型上测试,最佳整体得分仅为0.2847,表明多模态上下文学习尚未饱和。InternVL3.5-30B-A3B在上下文定位与新知识学习上表现最优,而Qwen3.5-Plus在新信息应用上领先。我们还分析了判别者可靠性、上下文长度、图像数量及典型失败案例。代码已开源。

原文摘要 · Abstract (English)

Real-world tasks often require models to learn from task-specific context rather than relying only on pre-trained knowledge. While recent work has highlighted this capability as context learning, existing evaluations mainly focus on textual contexts. In many practical settings, however, the context to be learned from is multimodal: scientific findings are conveyed through figures and tables, financial indicators are scattered across converted reports, and spatial decisions depend on maps, scenes, or web pages. We introduce CLBench-V, a benchmark for multimodal context learning that addresses the difficulty of localizing where context use breaks down by organizing tasks around three dimensions: context grounding, new information application, and new knowledge learning. CLBench-V combines converted public benchmarks with newly constructed datasets spanning domains such as science, finance, long-document understanding, spatial reasoning, and web-based visual question answering. To reduce the cost of constructing domain-specific context-learning tasks, we further use automated construction and filtering procedures for our newly built datasets. Across 3,443 instances and six recent multimodal models, the best overall score is only 0.2847, indicating that multimodal context learning remains far from saturated. Moreover, InternVL3.5-30B-A3B performs best on context grounding and new knowledge learning, while Qwen3.5-Plus performs best on new information application. We further analyze judge reliability, context length, image count, and representative failure cases. Code is available at https://github.com/IamLihua/CLBench-V.

多模态上下文学习评测基准知识获取

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。