arXiv:2605.09188cs.LGcs.AI2026-05被引 3

让AI学会按难易程度调整回答策略,提升训练与推理效率。

DARE: Difficulty-Adaptive Reinforcement Learning with Co-Evolved Difficulty Estimation

论文配图:DARE: Difficulty-Adaptive Reinforcement Learning with Co-Evolved Difficulty Estimation
图 1 · 摘自论文原文
  • 通过自适应难度估计与策略协同进化,动态优化训练重点。
  • 在多个模型上实现更高准确率,同时减少简单任务的冗余输出。
  • 适合追求高效强化学习和推理性能优化的研究者与工程师。

强化学习能提升大语言模型的推理能力,但成本高且样本效率低,因大量生成序列提供弱学习信号。现有难度感知数据选择方法虽优先处理中等难度提示,但分析发现三方面局限:策略漂移下难度估计失准、仅数据筛选带来的最终性能提升有限、推理效率未显著改善。这表明,高效有效的强化学习需超越难度过滤:策略应学会解决难题,同时对简单任务生成简洁响应。为此,我们提出 **Dare**——一个通过自归一化重要性采样协同进化难度估计与策略的统一框架,利用对称贝塔采样分布保持难度多样性,并针对不同难度层级实施自适应计算分配的定制化训练策略。在多个模型与领域上的大量实验表明,**Dare** 在训练效率、最终效果及推理效率上均优于现有方法,对简单任务生成更简洁回答,同时提升复杂任务的正确性。代码已开源:https://github.com/EtaYang10th/DARE。

原文摘要 · Abstract (English)

Reinforcement learning improves the reasoning ability of large language models but remains costly and sample-inefficient, as many rollouts provide weak learning signals. Difficulty-aware data selection methods attempt to address this by prioritizing moderately difficult prompts, yet our analysis reveals three limitations: difficulty estimates become inaccurate under policy drift, data selection alone yields limited final-performance gains, and inference efficiency remains largely unchanged. These findings suggest that efficient and effective RL requires more than filtering by difficulty: the policy should learn to solve hard tasks while producing concise responses for easy ones. To this end, we propose **Dare**, a unified framework that co-evolves difficulty estimation with the policy via self-normalized importance sampling, maintains diverse difficulty coverage through a symmetric Beta sampling distribution, and applies tailored training strategies across difficulty tiers with adaptive compute allocation. Extensive experiments across multiple models and domains demonstrate that **Dare** consistently outperforms existing methods in training efficiency, final effectiveness, and inference efficiency, producing more concise responses on easy tasks while improving correctness on hard ones. Code is available at https://github.com/EtaYang10th/DARE.

强化学习大模型难度自适应推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。