让大模型通过互动学习,提升独立解题能力。
Interactive Learning for LLM Reasoning
- 设计动态交互机制,根据题目难度自动切换合作或竞争策略。
- 在5个基准上实现最高5%的性能提升,强模型鲁棒性增强。
- 适合研究多智能体协作与大模型推理能力提升的学者。
现有基于多智能体的学习方法构建了交互训练环境以促进多个大语言模型(LLMs)间的协作,从而构建更强的多智能体系统(MAS)。然而,在推理阶段,这些方法需要重新执行整个系统才能获得最终答案,这与人类认知不符——个体可通过与他人互动提升自身推理能力,并在未来独立解决问题。为探究多智能体互动能否增强LLMs的独立解题能力,我们提出ILR,一种新型共学习框架,包含两个关键组件:动态交互与感知校准。具体而言,动态交互首先根据题目难度和模型能力自适应选择合作或竞争策略;随后,模型通过Idea3这一创新交互范式进行信息交换,模拟人类讨论过程,再各自得出最终答案。在感知校准中,ILR采用组相对策略优化(GRPO),将一个模型的奖励分布特征融入另一个模型的奖励函数,从而增强多智能体间协作的一致性。我们在两个模型家族中选取三个不同规模的LLM,在五个数学、一个编码、一个通用问答和一个科学推理基准上评估ILR的有效性。实验结果表明,ILR始终优于单智能体学习,相比最强基线最高提升达5%。进一步发现,Idea3能增强强模型在多智能体推理中的鲁棒性,而动态交互类型相比纯合作或纯竞争策略更能提升多智能体学习效果。
原文摘要 · Abstract (English)
Existing multi-agent learning approaches have developed interactive training environments to explicitly promote collaboration among multiple Large Language Models (LLMs), thereby constructing stronger multi-agent systems (MAS). However, during inference, they require re-executing the MAS to obtain final solutions, which diverges from human cognition that individuals can enhance their reasoning capabilities through interactions with others and resolve questions independently in the future. To investigate whether multi-agent interaction can enhance LLMs' independent problem-solving ability, we introduce ILR, a novel co-learning framework for MAS that integrates two key components: Dynamic Interaction and Perception Calibration. Specifically, Dynamic Interaction first adaptively selects either cooperative or competitive strategies depending on question difficulty and model ability. LLMs then exchange information through Idea3, an innovative interaction paradigm designed to mimic human discussion, before deriving their respective final answers. In Perception Calibration, ILR employs Group Relative Policy Optimization (GRPO) to train LLMs while integrating one LLM's reward distribution characteristics into another's reward function, thereby enhancing the cohesion of multi-agent interactions. We evaluate the effectiveness of ILR across three LLMs from two model families of varying scales on five mathematical, one coding, one general question answering, and one scientific reasoning benchmarks. Experimental results show that ILR consistently outperforms single-agent learning, yielding an improvement of up to 5% over the strongest baseline. We further discover that Idea3 can enhance the robustness of stronger LLMs during multi-agent inference, and dynamic interaction types can boost multi-agent learning compared to pure cooperative or competitive strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。