用小模型实现高精度表格问答,成本低效果好。
Accurate Table Question Answering with Accessible LLMs
- 拆解复杂问题为多个简单任务,由不同智能体协作完成
- 在WikiTQ上用140亿参数模型达72.1%准确率,接近GPT-4表现
- 适合想低成本使用大模型能力的研究者和开发者
给定数据库中的表格T和自然语言问题Q,表格问答(TQA)任务旨在基于T的内容返回准确答案。近期先进方法依赖需付费的大型专有大模型,带来高昂成本。本文聚焦于使用可本地运行的小型开源大模型进行TQA,这类模型虽能力较弱,但能显著降低使用门槛。我们发现,性能下降主因是以往方法要求模型处理长而复杂的提示,超出小型模型承受能力。为此提出Orchestra——一种多智能体协同框架,通过分层结构将复杂任务分解为多个简单子任务,各智能体专注单一职责,如同交响乐团配合。该设计大幅降低单个模型的提示复杂度,提升输出可靠性。我们在AgentScope框架上实现Orchestra,使用多种开源大模型在多个TQA基准上评估。结果表明,即使在小至中等规模模型上,Orchestra也表现优异:以Qwen2.5-14B在WikiTQ上达到72.1%准确率,接近此前使用GPT-4获得的75.3%最优结果;使用更大模型如Qwen、Llama或DeepSeek时,其性能超越所有已有方法,在所有基准上建立新SOTA。
原文摘要 · Abstract (English)
Given a table T in a database and a question Q in natural language, the table question answering (TQA) task aims to return an accurate answer to Q based on the content of T. Recent state-of-the-art solutions leverage large language models (LLMs) to obtain high-quality answers. However, most rely on proprietary, large-scale LLMs with costly API access, posing a significant financial barrier. This paper instead focuses on TQA with smaller, open-weight LLMs that can run on a desktop or laptop. This setting is challenging, as such LLMs typically have weaker capabilities than large proprietary models, leading to substantial performance degradation with existing methods. We observe that a key reason for this degradation is that prior approaches often require the LLM to solve a highly sophisticated task using long, complex prompts, which exceed the capabilities of small open-weight LLMs. Motivated by this observation, we present Orchestra, a multi-agent approach that unlocks the potential of accessible LLMs for high-quality, cost-effective TQA. Orchestra coordinates a group of LLM agents, each responsible for a relatively simple task, through a structured, layered workflow to solve complex TQA problems -- akin to an orchestra. By reducing the prompt complexity faced by each agent, Orchestra significantly improves output reliability. We implement Orchestra on top of AgentScope, an open-source multi-agent framework, and evaluate it on multiple TQA benchmarks using a wide range of open-weight LLMs. Experimental results show that Orchestra achieves strong performance even with small- to medium-sized models. For example, with Qwen2.5-14B, Orchestra reaches 72.1% accuracy on WikiTQ, approaching the best prior result of 75.3% achieved with GPT-4; with larger Qwen, Llama, or DeepSeek models, Orchestra outperforms all prior methods and establishes new state-of-the-art results across all benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。