arXiv:2506.08989cs.LGcs.CL2025-06NeurIPS被引 36

让大模型自己发现弱点并生成针对性题目,提升推理能力。

SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning

  • 模型自识别错误题目,提取核心概念生成新题
  • 7B和32B模型在8个基准上分别提升10.0%和7.7%
  • 无需外部知识,适合强化学习训练复杂推理任务

基于可验证奖励的强化学习(RLVR)在训练大语言模型解决复杂推理任务(如数学问题)方面已证明有效。其可扩展性的前提是高质量的问题集与精确可验证的答案。然而,现有基于蒸馏的合成数据集存在人工标注数学题稀缺、验证答案有限的问题,限制了其在强化学习中的效果。此外,多数问题生成策略盲目扩充题库,未考虑模型能力,导致生成效率低下。为此,我们提出自知缺陷驱动的问题生成框架(SwS),系统识别模型缺陷,并据此增强问题集。具体而言,将模型在强化学习迭代采样中反复出错的问题定义为弱点,从中提取核心概念,生成新题以强化模型薄弱环节。该框架不依赖外部知识蒸馏,使模型在强化学习中自主识别并克服自身弱点,实现7B和32B模型在八个主流推理基准上的平均性能提升10.0%和7.7%。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for training large language models (LLMs) on complex reasoning tasks, such as mathematical problem solving. A prerequisite for the scalability of RLVR is a high-quality problem set with precise and verifiable answers. However, the scarcity of well-crafted human-labeled math problems and limited-verification answers in existing distillation-oriented synthetic datasets limit their effectiveness in RL. Additionally, most problem synthesis strategies indiscriminately expand the problem set without considering the model's capabilities, leading to low efficiency in generating useful questions. To mitigate this issue, we introduce a Self-aware Weakness-driven problem Synthesis framework (SwS) that systematically identifies model deficiencies and leverages them for problem augmentation. Specifically, we define weaknesses as questions that the model consistently fails to learn through its iterative sampling during RL training. We then extract the core concepts from these failure cases and synthesize new problems to strengthen the model's weak areas in subsequent augmented training, enabling it to focus on and gradually overcome its weaknesses. Without relying on external knowledge distillation, our framework enables robust generalization byempowering the model to self-identify and address its weaknesses in RL, yielding average performance gains of 10.0% and 7.7% on 7B and 32B models across eight mainstream reasoning benchmarks.

强化学习大模型推理问题生成自适应训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。