用几何轨迹找关键层,13层适配胜过全模型微调
RDP LoRA: Geometry-Driven Identification for Parameter-Efficient Adaptation in Large Language Models

- 通过几何轨迹简化算法识别表示变化的关键层
- 仅用13个选定层在MMLU-Math上达81.67%准确率
- 无需训练即可指导高效微调,适合资源受限场景
尽管低秩适应(LoRA)等参数高效方法广泛应用,但大型语言模型内部表征的层间角色仍不明确,导致适配位置选择依赖经验。本文将隐藏状态演化建模为高维几何轨迹,采用无需参数、无需训练的拉默-道格拉斯-佩克(RDP)算法,保留全局结构转变的同时剔除局部冗余变化,识别出表示路径上的关键转折点。这些几何拐点不仅用于分析,更直接作为微调时层选择的决策信号。将该几何感知层选择策略融入Qwen3-8B-Base的LoRA微调,在仅使用13个RDP选定层的情况下,于MMLU-Math任务上达到81.67%准确率,显著优于全模型36层微调(79.32%)、随机选13层(75.56%)及基线模型(74.25%)。结果表明,利用表征轨迹内在几何特性可提供稳健、可解释且无需训练的层选择信号。
原文摘要 · Abstract (English)
Fine-tuning Large Language Models (LLMs) remains structurally uncertain despite parameter-efficient methods such as Low-Rank Adaptation (LoRA), as the layer-specific roles of internal representations are poorly understood, leading to heuristic decisions about where adaptation should be applied. We model the evolution of hidden states as a high-dimensional geometric trajectory and propose using the Ramer-Douglas-Peucker (RDP) algorithm, a parameter-free and training-free polygon simplification method that preserves global structural transitions while eliminating locally redundant changes, to identify critical breakpoints along the representation path. Crucially, we use these geometric pivots not merely for analysis, but as a direct decision signal for determining which layers should be adapted during parameter-efficient fine-tuning. By integrating this geometry-aware layer selection strategy into LoRA fine-tuning of Qwen3-8B-Base, we achieve superior performance on MMLU-Math using only 13 RDP-selected layers (81.67%), significantly outperforming both full 36-layer adaptation (79.32%) and random 13-layer selection (75.56%), as well as the baseline Qwen3-8B-Base model (74.25%). These results demonstrate that leveraging the intrinsic geometry of representation trajectories provides a robust, interpretable, and training-free signal for optimizing layer selection during model adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。