通过任务映射优化多模态上下文学习序列，提升大模型推理能力
TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configuration
- 基于任务映射动态调整输入序列结构
- 在9个数据集上显著超越现有基线方法
- 适合需要复杂推理的多模态应用开发者
多模态上下文学习（ICL）是发挥大视觉语言模型（LVLMs）能力的关键机制，但其效果高度依赖输入序列质量，尤其在涉及复杂推理或开放式生成的任务中。当前对LVLM如何在推理过程中利用这些序列的理解有限。本文从任务映射视角系统解析多模态ICL，揭示演示样本内与之间的局部和全局关系如何引导模型推理。基于此，提出TACO——一个轻量级Transformer模型，具备任务感知注意力，可动态配置ICL序列。通过将任务映射信号注入自回归解码过程，实现序列构建与任务推理的双向协同。在五个LVLM和九个数据集上的实验表明，TACO在多种ICL任务中持续优于基线。结果表明任务映射为理解与改进多模态ICL提供了新视角。
原文摘要 · Abstract (English)
Multimodal in-context learning (ICL) has emerged as a key mechanism for harnessing the capabilities of large vision-language models (LVLMs). However, its effectiveness remains highly sensitive to the quality of input ICL sequences, particularly for tasks involving complex reasoning or open-ended generation. A major limitation is our limited understanding of how LVLMs actually exploit these sequences during inference. To bridge this gap, we systematically interpret multimodal ICL through the lens of task mapping, which reveals how local and global relationships within and among demonstrations guide model reasoning. Building on this insight, we present TACO, a lightweight transformer-based model equipped with task-aware attention that dynamically configures ICL sequences. By injecting task-mapping signals into the autoregressive decoding process, TACO creates a bidirectional synergy between sequence construction and task reasoning. Experiments on five LVLMs and nine datasets demonstrate that TACO consistently surpasses baselines across diverse ICL tasks. These results position task mapping as a novel and valuable perspective for interpreting and improving multimodal ICL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。