揭示多模态上下文学习中模态间的不对称性与机制
Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformers
- 通过小规模模型控制实验，研究跨模态上下文学习机制
- 主模态高多样性时，辅模态低复杂度即可触发多模态ICL
- 发现标签复制是核心机制，适用于理解大模型推理过程
基于Transformer的多模态大语言模型常表现出上下文学习（ICL）能力。我们探究：现代Transformer如何从上下文示例中学习跨模态信息关联？通过在合成分类任务上训练的小型Transformer进行受控实验，可精确操控数据统计与模型结构。首先复现单模态ICL核心原理，发现旋转位置编码（RoPE）提高了ICL的数据复杂度阈值。扩展至多模态场景后，揭示根本性学习不对称性：当模型在主模态的高多样性数据上预训练时，辅模态仅需极低数据复杂度即可触发多模态ICL。机制分析表明，两种情形均依赖于类归纳机制——从匹配的上下文示例中复制标签；多模态训练则将此类电路跨模态细化与拓展。研究为理解现代Transformer中的多模态ICL提供了机制基础，并构建了可控测试平台。代码已开源。
原文摘要 · Abstract (English)
Transformer-based multimodal large language models often exhibit in-context learning (ICL) abilities. Motivated by this phenomenon, we ask: how do transformers learn to associate information across modalities from in-context examples? We investigate this question through controlled experiments on small transformers trained on synthetic classification tasks, enabling precise manipulation of data statistics and model architecture. We begin by revisiting core principles of unimodal ICL in modern transformers. While several prior findings replicate, we find that Rotary Position Embeddings (RoPE) increases the data complexity threshold for ICL. Extending to the multimodal setting reveals a fundamental learning asymmetry: when pretrained on high-diversity data from a primary modality, surprisingly low data complexity in the secondary modality suffices for multimodal ICL to emerge. Mechanistic analysis shows that both settings rely on an induction-style mechanism that copies labels from matching in-context exemplars; multimodal training refines and extends these circuits across modalities. Our findings provide a mechanistic foundation for understanding multimodal ICL in modern transformers and introduce a controlled testbed for future investigation. Code is available at: https://github.com/YiranHuangIrene/multimodal-icl
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。