用多个专家代理迭代修正大模型物理题的错误，提升准确率最多16%。
Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents
- 设计多代理系统，逐轮纠正理解偏差、概念误用和计算错误
- 在PhysicsQA等数据集上，大模型准确率最高提升16%
- 利用GPT-4o识别错误，让开源模型逼近其推理能力
大型语言模型在多种推理任务中表现出色，但在物理等科学推理任务中仍面临显著挑战，需兼具数学运算、事实理解和概念把握。面对复杂物理问题，现有模型常出现理解偏差、概念误用和计算错误。尽管可分别应对这些问题，但缺乏能同时解决三者的通用方法。为此，我们提出混合精炼代理（MoRA），一种新型代理式精炼框架，通过迭代修正模型生成的初始解，有效纠正上述错误，显著提升开源大模型性能。该方法借助GPT-4o作为错误识别器，指导多个精炼代理进行优化。我们在SciEval、MMLU及自建物理数据集PhysicsQA上评估，MoRA使Llama-3-70B与Gemma-2-27B在多项任务中准确率最高提升16%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) demonstrate remarkable capabilities in various reasoning tasks. However, they encounter significant challenges when it comes to scientific reasoning, particularly in physics, which requires not only mathematical reasoning but also factual and conceptual understanding. When addressing complex physics problems, LLMs typically face three key issues: problem miscomprehension, incorrect concept application, and computational errors. While each of these problems can be addressed individually, there is a need for a generalized approach that can tackle all three issues simultaneously. To address this, we introduce Mixture of Refinement Agents (MoRA), a novel agentic refinement framework that iteratively refines the LLM generated base solution by correcting the aforementioned errors, resulting in a significant performance improvement for open-source LLMs. Our approach aims to bridge the gap between opensource LLMs and GPT-4o by utilizing the latter as error identifier to guide these refinement agents. We evaluate our approach on the SciEval and MMLU subsets along with our own physics dataset (PhysicsQA). MoRA significantly improves the performance of Llama-3-70B and Gemma-2-27B on these datasets, achieving up to a 16% increase in final answer accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。