基于曲率与信任域的高效微调方法，提升大模型训练稳定性和效率。
CTR-LoRA: Curvature-Aware and Trust-Region Guided Low-Rank Adaptation for Large Language Models
- 根据曲率代理动态分配低秩参数，实现更优更新策略。
- 在7B-13B模型上显著提升准确率，同时降低内存和延迟。
- 适合资源受限场景下的大模型高效微调，尤其关注稳定性与性能平衡。
参数高效微调（PEFT）已成为在计算与内存受限条件下适配大语言模型的标准方法。尽管以往方法通过低秩更新、量化或启发式预算重分配提升了效率，但常将容量分配与训练中更新演化过程解耦。本文提出CTR-LoRA，一种由曲率信任域引导的框架，将秩调度与稳定性感知优化相结合。该方法基于轻量级二阶代理推导边际效用进行参数分配，并采用Fisher/Hessian度量的信任域约束更新。在多个开源骨干模型（7B-13B）上，针对分布内与分布外基准的实验表明，其性能持续优于强基线。除了精度提升，CTR-LoRA还增强了训练稳定性，降低内存开销，实现更高吞吐，位于性能与效率的帕累托前沿。结果揭示了更鲁棒、可部署的PEFT的系统性路径。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) has become the standard approach for adapting large language models under limited compute and memory budgets. Although previous methods improve efficiency through low-rank updates, quantization, or heuristic budget reallocation, they often decouple the allocation of capacity from the way updates evolve during training. In this work, we introduce CTR-LoRA, a framework guided by curvature trust region that integrates rank scheduling with stability-aware optimization. CTR-LoRA allocates parameters based on marginal utility derived from lightweight second-order proxies and constrains updates using a Fisher/Hessian-metric trust region. Experiments on multiple open-source backbones (7B-13B), evaluated on both in-distribution and out-of-distribution benchmarks, show consistent improvements over strong PEFT baselines. In addition to increased accuracy, CTR-LoRA enhances training stability, reduces memory requirements, and achieves higher throughput, positioning it on the Pareto frontier of performance and efficiency. These results highlight a principled path toward more robust and deployable PEFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。