针对强化学习推理的低秩微调方法,保留预训练模型结构并提升效率。
GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR
- 利用SVD提取强化学习更新子空间主方向,初始化低秩适配器
- 在1.5B到32B参数模型上,数学、医疗、编程任务均优于现有基线
- 适合需要结构保持和跨领域泛化的大型推理模型微调场景
强化学习与可验证奖励(RLVR)是提升大规模推理模型的关键范式。与监督微调(SFT)不同,RLVR具有独特的优化动态,对预训练几何结构的保持敏感。现有参数高效方法在此场景下存在局限:低秩适配方法如PiSSA主要为SFT设计,未考虑RLVR的独特优化特性与几何结构;而直接微调RLVR偏好的非结构化稀疏参数子空间则在现代硬件上面临效率瓶颈。为此,我们提出GeoRA(Geometry-Aware Low-Rank Adaptation),一种专为RLVR设计的低秩适配方法。GeoRA利用RL更新子空间的各向异性和可压缩性,通过奇异值分解(SVD)提取其主方向,初始化低秩适配器,同时冻结剩余组件作为结构锚点。该设计既保持了预训练结构,又支持高效密集计算。在Qwen和Llama系列模型(1.5B至32B参数)上的实验表明,GeoRA在数学、医学、编码等RLVR任务中持续优于强基线,且在域外任务上展现更强泛化能力与更少遗忘。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) is a key paradigm for improving large-scale reasoning models. Unlike supervised fine-tuning (SFT), RLVR exhibits distinct optimization dynamics and is sensitive to the preservation of pre-trained geometric structures. However, existing parameter-efficient methods face key limitations in this regime. Low-rank adaptation methods, such as PiSSA, are primarily designed for Supervised Fine-Tuning (SFT) and do not account for the distinct optimization dynamics and geometric structures of RLVR. Conversely, directly fine-tuning the unstructured sparse parameter subspace favored by RLVR encounters efficiency bottlenecks on modern hardware. To address these challenges, we propose GeoRA (Geometry-Aware Low-Rank Adaptation), a low-rank adaptation method tailored for RLVR. Specifically, GeoRA exploits the anisotropic and compressible structure of RL update subspace, and extracts its principal directions via Singular Value Decomposition (SVD) to initialize low-rank adapters, while freezing residual components as a structural anchor during training. This design preserves the pre-trained structure and enables efficient dense computation. Experiments on Qwen and Llama models from 1.5B to 32B parameters show that GeoRA consistently outperforms strong low-rank baselines across RLVR settings in mathematics, medicine, and coding, while showing stronger generalization and less forgetting on out-of-domain tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。