优化克罗内克适配器组件设计，提升大模型微调性能
Diving into Kronecker Adapters: Component Design Matters
- 分析组件维度与数量对适配器能力的影响
- 提出新配置使性能接近全量微调
- 提供可落地的参数预算配置指南
克罗内克适配器作为大模型微调的新兴方法，可通过可调组件结构实现高秩更新。然而现有研究多将组件结构视为固定或启发式设计，未充分探索其维度与数量的影响。本文首次将组件结构识别为决定适配器容量的关键因素，通过细粒度分析组件维度与数量，发现克罗内克适配器与全量微调的对齐程度依赖于具体配置。基于此，我们提出组件设计型克罗内克适配器（CDKA），并提供参数预算感知的配置指南与专用训练稳定策略，便于实际部署。在多种架构与模态上的实验验证了其有效性。代码已公开于 https://github.com/rainstonee/CDKA。
原文摘要 · Abstract (English)
Kronecker adapters have emerged as a promising approach for fine-tuning large-scale models, enabling high-rank updates through tunable component structures. However, existing work largely treats the component structure as a fixed or heuristic design choice, leaving the dimensions and number of Kronecker components underexplored. In this paper, we identify component structure as a key factor governing the capacity of Kronecker adapters. We perform a fine-grained analysis of both the dimensions and number of Kronecker components. In particular, we show that the alignment between Kronecker adapters and full fine-tuning depends on component configurations. Guided by these insights, we propose Component Designed Kronecker Adapters (CDKA). We further provide parameter-budget-aware configuration guidelines and a tailored training stabilization strategy for practical deployment. Experiments across various architectures and modalities demonstrate the effectiveness of CDKA. Code is available at https://github.com/rainstonee/CDKA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。