让嵌入向量在不同计算资源下都高效可用,提升小模型性能。
MIPIC: Matryoshka Representation Learning via Self-Distilled Intra-Relational and Progressive Information Chaining
- 通过自蒸馏对齐不同维度的语义关系,保证结构一致。
- 逐层迁移深层语义,使浅层模型也能学到高级特征。
- 适用于从极小到大型模型,尤其适合资源受限场景。
表示学习是自然语言处理的基础,但构建适用于不同计算预算的嵌入表示仍具挑战性。马特罗什卡表示学习(MRL)通过嵌套嵌入提供灵活的推理范式,但学习此类结构需显式协调信息在嵌入维度与模型深度间的分布。本文提出MIPIC(基于自蒸馏内部关联对齐与渐进信息链的马特罗什卡表示学习),一个统一训练框架,旨在生成结构连贯且语义紧凑的马特罗什卡表示。MIPIC通过自蒸馏内部关联对齐(SIA)促进跨维度结构一致性,利用top-k CKA自蒸馏对齐完整与截断表示之间的令牌级几何与注意力驱动关系;同时,通过渐进信息链(PIC)实现深度间语义凝聚,逐步将深层成熟的任务语义传递至浅层。在STS、NLI及分类基准上的大量实验(涵盖TinyBERT、BGEM3、Qwen3等模型)表明,MIPIC生成的马特罗什卡表示在所有容量下均表现优异,尤其在极端低维条件下展现出显著性能优势。
原文摘要 · Abstract (English)
Representation learning is fundamental to NLP, but building embeddings that work well at different computational budgets is challenging. Matryoshka Representation Learning (MRL) offers a flexible inference paradigm through nested embeddings; however, learning such structures requires explicit coordination of how information is arranged across embedding dimensionality and model depth. In this work, we propose MIPIC (Matryoshka Representation Learning via Self-Distilled Intra-Relational Alignment and Progressive Information Chaining), a unified training framework designed to produce structurally coherent and semantically compact Matryoshka representations. MIPIC promotes cross-dimensional structural consistency through Self-Distilled Intra-Relational Alignment (SIA), which aligns token-level geometric and attention-driven relations between full and truncated representations using top-k CKA self-distillation. Complementarily, it enables depth-wise semantic consolidation via Progressive Information Chaining (PIC), a scaffolded alignment strategy that incrementally transfers mature task semantics from deeper layers into earlier layers. Extensive experiments on STS, NLI, and classification benchmarks (spanning models from TinyBERT to BGEM3, Qwen3) demonstrate that MIPIC yields Matryoshka representations that are highly competitive across all capacities, with significant performance advantages observed under extreme low-dimensional.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。