通过挖掘隐含逻辑提升大模型自我推理能力,解决错误累积问题。
LC-ERD: Mining Latent Logic for Self-Evolving Reasoning via Consistency-Regulated Reward Decomposition

- 从模型自身提炼隐含逻辑专家共识,构建变分逻辑势能
- 多智能体分解奖励,精准评估每一步推理价值
- 可发现传统奖励忽略的高质量推理模式,适合自进化研究者
大语言模型推理能力的提升受限于高质量过程数据的稀缺。尽管基于内生奖励的自对齐提供了解决方案,但有效监督的挖掘面临三大挑战:(1) 模仿偏差导致的标签噪声,奖励偏好统计似然而非逻辑真实,形成“正确性幻觉”掩盖错误累积;(2) 粗粒度监督,稀疏全局结果(如GRPO)无法提供细粒度引导,将推理链视为整体;(3) 分布崩溃,信号缺乏泛化能力,加剧预训练偏差。为此,我们提出LC-ERD(逻辑一致性内生奖励分解)框架,将自对齐视为隐式结构挖掘。通过聚合模型隐含逻辑专长(LLE)的一致性,构建变分逻辑势能以去噪推理流形,并引入基于IGM原理的多智能体价值分解协议,量化每一步推理的效用。实验表明,LC-ERD实现稳健的自进化路径,揭示了逻辑一致性与准确率间的权衡,识别出标准奖励遗漏的高价值推理模式。代码已开源:https://github.com/LC-ERD-repo/LC-ERD。
原文摘要 · Abstract (English)
The evolution of Large Language Model (LLM) reasoning is bottlenecked by the scarcity of high-quality process data. While self-alignment via endogenous rewards offers a solution, mining valid supervision faces three challenges: (1) Label Noise via Mimetic Bias, where rewards prioritize statistical likelihood over logical truth, creating a "correctness illusion" that masks compounding errors; (2) Coarse-Grained Supervision, where sparse global outcomes (e.g., in GRPO) fail to provide granular guidance, treating reasoning chains as monolithic; and (3) Distributional Collapse, where signals fail to generalize without amplifying pre-training biases. To address these, we introduce LC-ERD (Logic-Consistent Endogenous Reward Decomposition), a framework framing self-alignment as latent structure mining. We derive a Variational Logic Potential by aggregating consensus from the model's Latent Logic Expertise (LLE) to denoise the reasoning manifold, and introduce a Multi-Agent Value Decomposition protocol based on the IGM principle to quantify individual step utility. Experiments show LC-ERD delivers a robust self-evolution path, uncovering trade-offs between logic consistency and accuracy while identifying high-value reasoning patterns missed by standard rewards. Our code is available at https://github.com/LC-ERD-repo/LC-ERD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。