rank-1微调不稳？新方法通过几何对齐解决初始方向问题。
When Is Rank-1 Enough? Geometry-Guided Initialization for Parameter-Efficient Fine-Tuning
- 基于视觉语言模态差异设计初始方向，让低秩更新更稳定
- 在多个任务上用rank-1达到rank-8效果，训练不崩溃
- 适合资源受限场景，尤其关注极低秩微调的研究者
参数高效微调（PEFT）是适配多模态大模型的常用方法，但极低秩设置（尤其是rank-1 LoRA）常不稳定。我们发现这种不稳定性并非仅因容量不足：在rank-1条件下，优化对更新方向极为敏感。预训练的视觉与文本特征形成不匹配的各向异性区域，产生主导的“模态间隙”方向，像平移分量一样显著影响早期梯度。分析表明，该模态间隙轴主导早期梯度流动，而随机rank-1初始化难以对齐此方向，导致梯度弱化和训练崩溃。为此提出Gap-Init，一种基于几何感知的初始化方法，将rank-1 LoRA方向对齐于小校准集估计的模态间隙向量，同时保持初始更新为零。在多个视觉-语言任务与骨干网络上，Gap-Init稳定了rank-1训练，性能可媲美甚至超越强基线rank-8。结果表明，在极端低秩极限下，初始对齐的重要性与秩本身相当。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) is a standard way to adapt multimodal large language models, yet extremely low-rank settings -- especially rank-1 LoRA -- are often unstable. We show that this instability is not solely due to limited capacity: in the rank-1 regime, optimization is highly sensitive to the update direction. Concretely, pretrained vision and text features form mismatched anisotropic regions, yielding a dominant "gap" direction that acts like a translation component and disproportionately steers early gradients under rank-1 constraints. Analyzing pretrained representations, we identify a modality-gap axis that dominates early gradient flow, while a random rank-1 initialization is unlikely to align with it, leading to weak gradients and training collapse. We propose Gap-Init, a geometry-aware initialization that aligns the rank-1 LoRA direction with an estimated modality-gap vector from a small calibration set, while keeping the initial LoRA update zero. Across multiple vision-language tasks and backbones, Gap-Init consistently stabilizes rank-1 training and can match or outperform strong rank-8 baselines. Our results suggest that at the extreme low-rank limit, initial alignment can matter as much as rank itself.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。