大模型用分步推理提效，小模型反而掉分，原因找到了。
Exploring Performance Contrasts in TableQA: Step-by-Step Reasoning Boosts Bigger Language Models, Limits Smaller Language Models
- 设计分步推理流程Table-Logic，逐层分析表格与问题
- 大模型（Llama-3-70B）准确率提升7.8%，小模型（Llama-2-7B）下降11%
- 揭示小模型在复杂推理中的瓶颈，适合研究模型能力边界者阅读
本文提出一种名为Table-Logic的详细提示流程，用于研究大模型与小模型在表格问答（TableQA）任务中使用分步推理时的性能差异。该方法通过依次识别问题相关的关键列与行、确定必要的聚合、计算或比较操作，最终推断并生成精确答案。在HybridQA数据集上，采用该方法后，大模型Llama-3-70B的准确率提升7.8%；而小模型Llama-2-7B的性能则下降11%。我们从多个维度实证分析了性能差异的成因，揭示了分步推理对小模型的局限性，并为改进小模型推理能力提供了潜在思路。
原文摘要 · Abstract (English)
This paper proposes a detailed prompting flow, termed Table-Logic, to investigate the performance contrasts between bigger and smaller language models (LMs) utilizing step-by-step reasoning methods in the TableQA task. The method processes tasks by sequentially identifying critical columns and rows given question and table with its structure, determining necessary aggregations, calculations, or comparisons, and finally inferring the results to generate a precise prediction. By deploying this method, we observe a 7.8% accuracy improvement in bigger LMs like Llama-3-70B compared to the vanilla on HybridQA, while smaller LMs like Llama-2-7B shows an 11% performance decline. We empirically investigate the potential causes of performance contrasts by exploring the capabilities of bigger and smaller LMs from various dimensions in TableQA task. Our findings highlight the limitations of the step-by-step reasoning method in small models and provide potential insights for making improvements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。