让大模型真正理解数学概念,而非仅套公式解题。
CORE: Concept-Oriented Reinforcement for Bridging the Definition-Application Gap in Mathematical Reasoning
- 用概念描述生成针对性测验,引导模型关注核心思想。
- 在推理过程中注入概念提示,提升解题时的思维连贯性。
- 适合想提升模型数学理解力的研究者和开发者。
大型语言模型(LLMs)虽能解决复杂数学题目,却常在需要真实理解时失败。现有基于可验证奖励的强化学习(RLVR)只优化最终答案,缺乏细粒度的概念信号,导致模型更擅长模式复用而非概念应用。本文提出CORE(概念导向强化),利用高质量教材中问题与概念描述的对应关系,先通过探针验证模型能复述定义但无法关联概念答题,量化概念推理差距。CORE随后:(i) 生成对齐概念的测验题;(ii) 在推理过程中注入简短概念片段,引导概念驱动的解题路径;(iii) 通过轨迹替换与轻量级前向KL约束,或直接在概念对齐测验上使用标准GRPO,强化概念推理。在多个模型上,CORE在域内概念-题目集及多样域外数学基准上均优于基线方法。该框架统一了直接训练与概念注入推演,在不依赖特定算法或验证器的前提下,实现了细粒度概念监督,弥合了解题能力与真实概念理解之间的鸿沟。
原文摘要 · Abstract (English)
Large language models (LLMs) often solve challenging math exercises yet fail to apply the concept right when the problem requires genuine understanding. Popular Reinforcement Learning with Verifiable Rewards (RLVR) pipelines reinforce final answers but provide little fine-grained conceptual signal, so models improve at pattern reuse rather than conceptual applications. We introduce CORE (Concept-Oriented REinforcement), an RL training framework that turns explicit concepts into a controllable supervision signal. Starting from a high-quality, low-contamination textbook resource that links verifiable exercises to concise concept descriptions, we run a sanity probe showing LLMs can restate definitions but fail concept-linked quizzes, quantifying the conceptual reasoning gap. CORE then (i) synthesizes concept-aligned quizzes, (ii) injects brief concept snippets during rollouts to elicit concept-primed trajectories, and (iii) reinforces conceptual reasoning via trajectory replacement after group failures, a lightweight forward-KL constraint that aligns unguided with concept-primed policies, or standard GRPO directly on concept-aligned quizzes. Across several models, CORE delivers consistent gains over vanilla and SFT baselines on both in-domain concept-exercise suites and diverse out-of-domain math benchmarks. CORE unifies direct training on concept-aligned quizzes and concept-injected rollouts under outcome regularization. It provides fine-grained conceptual supervision that bridges problem-solving competence and genuine conceptual reasoning, while remaining algorithm- and verifier-agnostic.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。