针对大模型推理的低资源强化学习方法，提升准确率并兼容轻量微调。
Token-Efficient RL for LLM Reasoning
- 仅选关键输出词进行优化，降低内存与计算开销。
- 在SVAMP上将准确率从46%提升至70%以上。
- 适合参数受限场景下的模型推理能力增强。
我们提出针对大语言模型（LLM）推理中严格内存与算力限制的强化学习策略，特别关注与LoRA微调的兼容性。基于早期策略梯度方法与基线减法，设计了无需评判器的算法，仅在少量有信息量的输出词上操作，以减少内存占用并稳定训练。提出S-GRPO（Group Relative Policy Optimization的随机变体）和T-SPMO（基于词级前缀匹配的细粒度信用分配方法）。在Qwen2-1.5B上应用后，SVAMP基准准确率从46%提升至70%以上，并在多位数乘法任务中表现优异。令人意外的是，全词段的GRPO在LoRA下未能超越基础模型，表明选择性词级优化可能在低参数训练中起到隐式正则化作用。
原文摘要 · Abstract (English)
We propose reinforcement learning (RL) strategies tailored for reasoning in large language models (LLMs) under strict memory and compute limits, with a particular focus on compatibility with LoRA fine-tuning. Building on early policy gradient methods with baseline subtraction, we design critic-free methods that operate on a small, informative subset of output tokens to reduce memory usage and stabilize training. We introduce S-GRPO, a stochastic variant of Group Relative Policy Optimization, and T-SPMO, a token-level prefix matching approach for fine-grained credit assignment. Applied to Qwen2-1.5B, our methods raise accuracy on the SVAMP benchmark from 46% to over 70% and show strong performance on multi-digit multiplication. Surprisingly, full-token GRPO under LoRA fails to improve over the base model, suggesting that selective token-level optimization may act as an implicit regularizer in low-parameter training regimes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。