MiCA通过优化低秩子空间提升大模型知识获取效率。
MiCA Learns More Knowledge Than LoRA and Full Fine-Tuning
- 聚焦模型中被忽视的低秩子空间进行微调
- 知识吸收效率最高提升5.9倍，参数量仅需LoRA的6-60%
- 适合资源有限但需高效注入新知识的场景
Minor Component Adaptation（MiCA）是一种新型参数高效微调方法，专为大型语言模型设计，重点在于适配模型表示中未被充分利用的子空间。与传统方法如低秩适应（LoRA）聚焦主导子空间不同，MiCA利用奇异值分解识别与最小奇异值相关的次要奇异向量所对应的子空间，并将微调过程中的参数更新限制在这些方向上。在优化训练超参数条件下，该策略可实现高达5.9倍的知识获取提升，同时保持极低的参数开销（仅为LoRA的6%-60%）。结果表明，将适应限制在次要奇异方向上，能更高效、更稳定地将新知识融入预训练语言模型。
原文摘要 · Abstract (English)
Minor Component Adaptation (MiCA) is a novel parameter-efficient fine-tuning method for large language models that focuses on adapting underutilized subspaces of model representations. Unlike conventional methods such as Low-Rank Adaptation (LoRA), which target dominant subspaces, MiCA leverages Singular Value Decomposition to identify subspaces related to minor singular vectors associated with the least significant singular values and constrains the update of parameters during fine-tuning to those directions. This strategy leads to up to 5.9x improvement in knowledge acquisition under optimized training hyperparameters and a minimal parameter footprint of 6-60% compared to LoRA. These results suggest that constraining adaptation to minor singular directions provides a more efficient and stable mechanism for integrating new knowledge into pre-trained language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。