让视觉语言模型更快地用示例学习,速度提升显著。
Parallel In-context Learning for Large Vision Language Models
- 将示例分块并行处理,降低计算开销。
- 性能接近完整上下文,但推理速度大幅提升。
- 适合需要快速适应新任务的实时应用。
大型视觉语言模型(LVLM)通过多模态上下文学习(MM-ICL)利用示例来适应新任务。虽然增加示例数量能提升性能,但会因Transformer注意力随上下文长度呈二次增长而带来显著推理延迟。为此,我们提出并行上下文学习(Parallel-ICL),一种即插即用的推理算法。Parallel-ICL将长示例上下文划分为多个较短、可管理的块,分块并行处理,并在logit层整合预测结果,采用加权产品专家(PoE)集成方法近似全上下文输出。基于集成学习理论,我们设计了两个策略:(i) 基于聚类的上下文分块以最大化块间多样性;(ii) 基于相似性的上下文合并,按查询相关性加权预测。在VQA、图像描述和分类基准上的大量实验表明,Parallel-ICL在保持与全上下文MM-ICL相当性能的同时,显著提升了推理速度。本工作为MM-ICL中的准确率-效率权衡提供有效解决方案,实现动态任务适应且推理开销大幅降低。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) employ multi-modal in-context learning (MM-ICL) to adapt to new tasks by leveraging demonstration examples. While increasing the number of demonstrations boosts performance, they incur significant inference latency due to the quadratic computational cost of Transformer attention with respect to the context length. To address this trade-off, we propose Parallel In-Context Learning (Parallel-ICL), a plug-and-play inference algorithm. Parallel-ICL partitions the long demonstration context into multiple shorter, manageable chunks. It processes these chunks in parallel and integrates their predictions at the logit level, using a weighted Product-of-Experts (PoE) ensemble to approximate the full-context output. Guided by ensemble learning theory, we introduce principled strategies for Parallel-ICL: (i) clustering-based context chunking to maximize inter-chunk diversity and (ii) similarity-based context compilation to weight predictions by query relevance. Extensive experiments on VQA, image captioning, and classification benchmarks demonstrate that Parallel-ICL achieves performance comparable to full-context MM-ICL, while significantly improving inference speed. Our work offers an effective solution to the accuracy-efficiency trade-off in MM-ICL, enabling dynamic task adaptation with substantially reduced inference overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。