小模型上，两步自修正比五角色协作更高效准确
Two Calls Beat Five Agents: Evaluating Multi-Agent Pipelines Against Self-Refinement for Local Language Models
- 用两步自修正替代多角色协作，降低计算开销
- 小模型上自修正达86.2%准确率，耗能仅为多角色的1/7.4
- 格式与设计细节比架构复杂度更重要，适合本地部署
多角色大模型系统通过分工提升推理能力，但主要在大型商用模型上评估。本研究将结构化多角色系统Parishad部署于Qwen2.5-7B-Instruct本地模型，在GSM8K（500题）和HumanEval（164题）上对比直接提示与两步自修正策略。使用JSON格式时，多角色系统在GSM8K上准确率从75.0%降至45.0%，因错误累积；改用纯文本格式后恢复至82.0%。两步自修正（V1）在GSM8K上达86.2%准确率，令牌消耗降低7.4倍。但在HumanEval上（直接提示已达96.3%），V1反而使准确率跌至66.5%。引入任务感知门控重设计（V2）后，HumanEval准确率保持在95.1%。结果表明，通信格式与实现细节比架构复杂度更关键，简单方法可媲美或超越多角色流水线。所有代码与数据已公开。
原文摘要 · Abstract (English)
Multi-agent LLM pipeline systems break down the task among multiple roles for better reasoning, but are benchmarked mainly with large-scale commercial models. In this study, we investigate Parishad, a structured multi-agent system involving five roles, by deploying it on Qwen2.5-7B-Instruct, a local model, on two datasets: GSM8K (500 questions) and HumanEval (164 questions), compared with prompting directly and two-call self-refinement. The multi-agent system drops GSM8K accuracy from 75.0\% to 45.0\% with JSON data format due to the error accumulation problem. With plaintext format, the accuracy is restored to 82.0\%. A two-call self-refinement strategy (V1) can achieve 86.2\% accuracy on GSM8K, with 7.4$\times$ lower token usage. However, the same V1 implementation on HumanEval---where direct accuracy is already 96.3\%---actively destroys performance (66.5\%). A task-aware gated redesign (V2) applied to HumanEval preserves accuracy at 95.1\%. Our results demonstrate that communication format and implementation details determine outcomes more than architectural complexity, and that simpler approaches match or outperform multi-agent pipelines for local 7B model deployment. All code and data are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。