通过重述问题提升大模型推理探索能力，解决奖励消失与思维模式单一问题。
Transformation-Augmented GRPO for Enhancing Exploration in Reasoning of Large Language Models
- 用问题重述生成多种等价表述，扩大模型响应多样性。
- 在四组大模型上显著提升竞赛级基准的pass@k指标，最高增4.97分。
- 仅用原数据量即达到其他方法2.5倍数据训练的效果，适合高效训练场景。
Group Relative Policy Optimization (GRPO) 已成为大语言模型中基于可验证奖励强化学习的主流方法，但存在梯度消失和多样性崩溃两大缺陷：当训练问题过易或过难时，所有采样回复获得相同奖励，导致梯度为零；同时模型趋向于坍缩到单一推理模式。本文提出 Transformation-Augmented GRPO (TA-GRPO)，通过自动生成语义等价的问题重述（改变措辞、格式与信息顺序）来缓解上述问题。由于重述改变了模型感知难度，将原始问题与重述版本的响应合并后，可获得混合奖励与更丰富的推理路径。TA-GRPO 在扩展响应集上联合计算优势，并统一重要性比例至原始问题，使模型能从更多样化的解题尝试中学习。在 Qwen3-1.7B、Qwen3-4B、Llama-3.2-1B、Llama-3.2-3B 四个模型上实验表明，TA-GRPO 在 AMC、OlympiadBench、AIME24、AIME25 等竞赛级基准以及 Minerva、GPQA-Diamond 等分布外基准上均持续提升 pass@$k$ 性能。尤其对 Qwen3-1.7B 与 Qwen3-4B，pass@32 平均分别提升 4.97 和 4.34 分，且仅用原数据量即达到基线需 2.5 倍数据训练的探索质量。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) has become the dominant method for reinforcement learning with verifiable rewards in large language models, but it suffers from two critical limitations: gradient vanishing and diversity collapse. When training questions are too easy or too hard, all sampled responses receive identical rewards, yielding zero gradients. Meanwhile, the model tends to collapse its responses toward a single reasoning pattern rather than exploring diverse strategies. We propose Transformation-Augmented GRPO (TA-GRPO), a simple but effective method that addresses both issues via question rephrasing. For each training question, we automatically generate multiple problem-equivalent rephrasings that alter wording, format, and information order while preserving the underlying meaning. Because these rephrasings shift the model's perceived difficulty, pooling responses across the original and its rephrasings yields mixed rewards and more diverse reasoning paths. TA-GRPO jointly computes advantages over this expanded response set and aligns all importance ratios to the original question, enabling the model to learn from a richer set of solution attempts. Experiments on four LLMs (Qwen3-1.7B, Qwen3-4B, Llama-3.2-1B, Llama-3.2-3B) show that TA-GRPO consistently improves pass@$k$ on competition-level benchmarks (AMC, OlympiadBench, AIME24, AIME25) and out-of-distribution benchmarks (Minerva, GPQA-Diamond). Notably, it improves the average pass@32 of Qwen3-1.7B and Qwen3-4B by \textbf{4.97} and \textbf{4.34} points, respectively, and matches the exploration quality of baselines trained on up to 2.5$\times$ more data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。