让大模型多次尝试答题并根据反馈改进，显著提升解题准确率。
Learning from Failures in Multi-Attempt Reinforcement Learning
- 设计多轮尝试任务，允许模型基于错误反馈迭代优化答案。
- 小模型在数学题上从1次尝试的45.6%提升至2次尝试的52.5%。
- 适合希望提升模型纠错与自我修正能力的研究者。
近期基于大语言模型（LLM）的强化学习进展，如DeepSeek R1，表明单次问答任务即可显著提升LLM的推理能力。本文将任务改为多轮尝试设置：模型对每个问题可进行多次回答，错误后获得反馈。该机制促使模型优化前序尝试，提高搜索效率。实验显示，仅在多轮任务上训练的小型模型，在数学基准测试中，尝试次数从1次增至2次时，准确率由45.6%升至52.5%；而标准单轮训练模型仅从42.3%微增至43.2%。结果表明，多轮训练的模型在数学任务上表现更优，且能更有效利用用户反馈进行自我修正。完整代码见https://github.com/DualityRL/multi-attempt。
原文摘要 · Abstract (English)
Recent advancements in reinforcement learning (RL) for large language models (LLMs), exemplified by DeepSeek R1, have shown that even a simple question-answering task can substantially improve an LLM's reasoning capabilities. In this work, we extend this approach by modifying the task into a multi-attempt setting. Instead of generating a single response per question, the model is given multiple attempts, with feedback provided after incorrect responses. The multi-attempt task encourages the model to refine its previous attempts and improve search efficiency. Experimental results show that even a small LLM trained on a multi-attempt task achieves significantly higher accuracy when evaluated with more attempts, improving from 45.6% with 1 attempt to 52.5% with 2 attempts on the math benchmark. In contrast, the same LLM trained on a standard single-turn task exhibits only a marginal improvement, increasing from 42.3% to 43.2% when given more attempts during evaluation. The results indicate that, compared to the standard single-turn task, an LLM trained on a multi-attempt task achieves slightly better performance on math benchmarks while also learning to refine its responses more effectively based on user feedback. Full code is available at https://github.com/DualityRL/multi-attempt
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。