模型自动生成简单问题变体，逐步提升解题能力。
LADDER: Self-Improving LLMs Through Recursive Problem Decomposition
- 通过递归生成更简单的问题变体来自我训练。
- 让小模型准确率从1%提升至82%，大模型达90%。
- 无需人工标注，适合想提升推理能力的开发者。
我们提出LADDER（自主难度驱动例题递归学习），一种使大语言模型通过递归生成并求解复杂问题的简化版本，实现自我驱动的能力提升。与依赖人工数据或反馈的方法不同，LADDER利用模型自身生成更易问题。在数学积分任务中，该方法将Llama 3.2 3B的准确率从1%提升至82%，使Qwen2.5 7B Deepseek-R1 Distilled在麻省理工积分选拔赛中达到73%。我们还引入TTRL（测试时强化学习），在推理阶段对测试问题的变体进行强化学习，使该模型在相同考试中取得90%的领先成绩，超越OpenAI o1。结果表明，无需架构扩展或人工监督，自我导向策略可显著增强模型能力。
原文摘要 · Abstract (English)
We introduce LADDER (Learning through Autonomous Difficulty-Driven Example Recursion), a framework which enables Large Language Models to autonomously improve their problem-solving capabilities through self-guided learning by recursively generating and solving progressively simpler variants of complex problems. Unlike prior approaches that require curated datasets or human feedback, LADDER leverages a model's own capabilities to generate easier question variants. We demonstrate LADDER's effectiveness in the subject of mathematical integration, improving Llama 3.2 3B's accuracy from 1% to 82% on undergraduate-level problems and enabling Qwen2.5 7B Deepseek-R1 Distilled to achieve 73% on the MIT Integration Bee qualifying examination. We also introduce TTRL (Test-Time Reinforcement Learning), where we perform reinforcement learning on variants of test problems at inference time. TTRL enables Qwen2.5 7B Deepseek-R1 Distilled to achieve a state-of-the-art score of 90% on the MIT Integration Bee qualifying examination, surpassing OpenAI o1's performance. These results show how self-directed strategic learning can achieve significant capability improvements without relying on architectural scaling or human supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。