用模型自信度调控自我进化，提升推理与数学能力
Confidence-Orchestrated Self-Evolution against Uncertain LLM Feedback
- 用模型自身置信度作为不确定信号，动态调节学习
- 在19个基准上平均表现优于基础模型，数学推理领先
- 适合追求自洽训练的AI研究者和开发者
自进化大语言模型通过自动生成训练任务和解法来减少对人工标注的依赖。但在许多推理领域，模型还需验证生成的任务并判断答案以获取训练信号。这带来训练信号挑战：错误的自我判断会导致错误的梯度更新。现有方法或依赖外部验证器（限制泛化性），或将噪声自生成反馈视为监督信号。本文提出COSE（置信度协调自进化），利用模型内在置信度作为轻量级不确定性信号来调节学习。COSE引入置信度加权PPO更新和置信度优先重放机制。在19个保留基准及四个Qwen/Llama主干（0.6B–4B）上，COSE持续优于基础模型，在通用推理和数学任务中达到最优平均表现，代码与数据见https://anonymous.4open.science/r/COSE_-B5C2。
原文摘要 · Abstract (English)
Self-evolving large language models (LLMs) learn by generating their own training tasks and solutions, reducing reliance on human-curated supervision. However, in many reasoning domains, the model must also validate generated tasks and judge generated answers to obtain training signals. This creates a training-signal challenge: erroneous self-judgments become erroneous gradient updates. Existing approaches either rely on external verifiers, which limits generality, or treat noisy self-generated feedback as supervision. We propose COSE (Confidence-Orchestrated Self-Evolution), which uses the LLM's intrinsic confidence as a lightweight uncertainty signal to modulate learning. COSE introduces confidence-weighted PPO updates and confidence-prioritized replay. Across 19 held-out benchmarks and four Qwen/Llama backbones (0.6B--4B), COSE consistently improves over base models and achieves the best average performance in general reasoning and mathematics, while remaining competitive on code. Code and data are available at https://anonymous.4open.science/r/COSE_-B5C2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。