提升多模态模型少样本学习精度，更真实模拟示范影响机制。
HIFICL: High-Fidelity In-Context Learning for Multimodal Tasks
- 用可学习的虚拟键值对模拟上下文，精准建模示范作用
- 在多个多模态任务上超越现有方法，提升显著且稳定
- 适合追求高精度少样本学习的开发者与研究者
上下文学习（ICL）是大型多模态模型（LMMs）的重要范式，通过少量上下文示范（ICDs）实现新任务适配。然而其性能对示范配置敏感且计算开销大。数学上，示范的影响可分解为标准注意力输出与上下文值的动态混合。现有近似方法通过学习一个“偏移向量”简化此过程。受精确分解启发，我们提出高保真上下文学习（HiFICL），更真实地建模ICL机制。HiFICL包含三个关键组件：1）一组“虚拟键值对”作为可学习上下文；2）低秩分解实现稳定正则化训练；3）简单端到端训练目标。从另一视角看，该机制构成一种上下文感知的参数高效微调（PEFT）。大量实验表明，HiFICL在多个多模态基准上持续优于现有近似方法。代码已开源：https://github.com/bbbandari/HiFICL。
原文摘要 · Abstract (English)
In-Context Learning (ICL) is a significant paradigm for Large Multimodal Models (LMMs), using a few in-context demonstrations (ICDs) for new task adaptation. However, its performance is sensitive to demonstration configurations and computationally expensive. Mathematically, the influence of these demonstrations can be decomposed into a dynamic mixture of the standard attention output and the context values. Current approximation methods simplify this process by learning a "shift vector". Inspired by the exact decomposition, we introduce High-Fidelity In-Context Learning (HIFICL) to more faithfully model the ICL mechanism. HIFICL consists of three key components: 1) a set of "virtual key-value pairs" to act as a learnable context, 2) a low-rank factorization for stable and regularized training, and 3) a simple end-to-end training objective. From another perspective, this mechanism constitutes a form of context-aware Parameter-Efficient Fine-Tuning (PEFT). Extensive experiments show that HiFICL consistently outperforms existing approximation methods on several multimodal benchmarks. The code is available at https://github.com/bbbandari/HiFICL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。