大模型能自己发明优化算法,实现自我进化。
Can Large Language Models Invent Algorithms to Improve Themselves?: Algorithm Discovery for Recursive Self-Improvement through Reinforcement Learning
- 让大模型自主生成、测试并优化改进自身的方法。
- 在数学推理上提升6%表现,优于人类设计方法4.3%。
- 发现的算法泛化强,跨模型适用无需重新训练。
大型语言模型(LLMs)已具备强大能力,但其改进仍受限于人工设计。本文提出Self-Developing框架,使LLMs能自主发现、实现并迭代优化自身的改进算法。该框架采用循环机制:初始模型生成可执行代码形式的算法候选,评估其效果,并通过直接偏好优化持续提升更复杂的改进策略。以模型合并为例,该框架成功发现新型合并算法,性能超越现有手工设计方法。在数学推理基准GSM8k上,自主发现算法使原始模型性能提升6%,超过任务算术(Task Arithmetic)等人类设计方法4.3%。令人惊讶的是,这些算法具有强泛化能力,在未参与优化的跨域模型上仍实现7.4%的增益。结果表明,大模型可突破训练限制,创造真正新颖的优化技术,迈向自主发展新阶段。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have achieved remarkable capabilities, yet their improvement methods remain fundamentally constrained by human design. We present Self-Developing, a framework that enables LLMs to autonomously discover, implement, and refine their own improvement algorithms. Our approach employs an iterative cycle where a seed model generates algorithmic candidates as executable code, evaluates their effectiveness, and uses Direct Preference Optimization to recursively improve increasingly sophisticated improvement strategies. We demonstrate this framework through model merging, a practical technique for combining specialized models. Self-Developing successfully discovered novel merging algorithms that outperform existing human-designed algorithms. On mathematical reasoning benchmarks, the autonomously discovered algorithms improve the seed model's GSM8k performance by 6\% and exceed human-designed approaches like Task Arithmetic by 4.3\%. Remarkably, these algorithms exhibit strong generalization, achieving 7.4\% gains on out-of-domain models without re-optimization. Our findings demonstrate that LLMs can transcend their training to invent genuinely novel optimization techniques. This capability represents a crucial step toward a new era where LLMs not only solve problems but autonomously develop the methodologies for their own advancement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。