无需示例也能稳定高效完成多模态推理,提升模型表现。
Hyper-ICL: Attention Calibration with Hyperbolic Anchor Distillation for Multimodal In-Context Learning

- 用低秩适配器校准注意力分布,模拟示例影响。
- 在六个基准上准确率显著优于基线方法。
- 适合追求高效、稳定多模态推理的开发者使用。
多模态上下文学习(Multimodal ICL)是一种实用的推理范式,通过少量图文交错的上下文示例(ICDs)引导多模态大模型解决新任务。然而,该方法存在推理延迟高、对示例格式、顺序和内容敏感等问题。为此,我们提出 Hyper-ICL,一种轻量级、基于训练的无示例多模态 ICL 框架,可在推理时直接重建示例效应,无需 ICD。Hyper-ICL 学习一个参数高效的低秩逻辑层适配器,校准注意力分布以匹配示例引发的注意力重分配。为捕捉示例影响随查询变化的差异,引入查询自适应调制机制,动态控制各层和头在词元级的干预强度。此外,提出逐层双曲锚点蒸馏损失,利用洛伦兹测地距离将学生中间特征与示例条件下的教师特征对齐,促使学生重构由 ICD 引起的示例-查询关系。在六个不同多模态基准(包括 VQAv2、OK-VQA、COCO Caption)上的大量实验表明,Hyper-ICL 在准确率和稳定性上均持续优于原始 ICL 及现有先进方法。
原文摘要 · Abstract (English)
Multimodal In-Context Learning (ICL) has emerged as a practical inference paradigm for Multimodal Large Language Models, where a small set of interleaved image-text In-Context Demonstrations (ICDs) conditions the model to solve new tasks. Despite its flexibility, multimodal ICL incurs high inference latency and suffers from instability due to sensitivity to demonstration formatting, ordering, and content. To address these limitations, we propose Hyper-ICL, a lightweight, training-based framework for demonstration-free multimodal ICL that reconstructs demonstration effects directly without requiring ICDs at inference time. Hyper-ICL learns a parameter-efficient low-rank logit-level adapter that calibrates attention distributions to better match demonstration-induced attention redistribution. To capture how demonstration influence varies across queries, we introduce a query-adaptive modulation mechanism that adaptively controls intervention strength at token level across layers and heads based on the current query. Finally, we propose a layer-wise hyperbolic anchor distillation loss that aligns intermediate student features to a demonstration-conditioned teacher via Lorentz geodesic distance. This loss encourages the student to reconstruct the demonstration-query relationships induced by ICDs. Extensive experiments across six different multimodal benchmarks (including VQAv2, OK-VQA, and COCO Caption) demonstrate that Hyper-ICL consistently improves accuracy and stability over vanilla ICL and existing state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。