优化提示词提升大模型表格事实验证能力,效果稳定且可推广。
Analyzing LLM Instruction Optimization for Tabular Fact Verification
- 用DSPy框架对比四种提示策略与三种优化器,系统评估指令优化效果。
- MiPROv2在CoT中提升最稳,SIMBA在大模型上对ReAct收益最大。
- 优化后推理更直接,减少冗余工具调用,适合小模型和高阶应用。
指令优化为增强大语言模型(LLM)推理能力提供了一种轻量、模型无关的方法。本文首次基于DSPy优化框架,系统比较了针对表格事实验证的指令优化方法。评估了四种开箱即用的提示技术:仅文本的直接预测、思维链(CoT)、使用SQL工具的ReAct,以及支持Python执行的CodeAct。研究涵盖三个来自DSPy框架的优化器——COPRO、MiPROv2和SIMBA,覆盖四个基准测试和三类模型。结果表明,指令优化能持续提升验证准确率;其中MiPROv2在CoT中表现最稳定,而SIMBA在大型模型上的ReAct代理中带来最大收益。行为分析显示,SIMBA通过引入启发式规则,促使推理路径更直接,增强了数值比较能力,并减少了ReAct代理中的非必要工具调用。在不同提示策略中,CoT在小模型上依然有效;尽管大模型构建的ReAct代理可达到竞争力表现,但需精细指令优化。
原文摘要 · Abstract (English)
Instruction optimization provides a lightweight, model-agnostic approach to enhancing the reasoning performance of large language models (LLMs). This paper presents the first systematic comparison of instruction optimization, based on the DSPy optimization framework, for tabular fact verification. We evaluate four out-of-the-box prompting techniques that cover both text-only prompting and code use: direct prediction, Chain-of-Thought (CoT), ReAct with SQL tools, and CodeAct with Python execution. We study three optimizers from the DSPy framework -- COPRO, MiPROv2, and SIMBA -- across four benchmarks and three model families. We find that instruction optimization consistently improves verification accuracy, with MiPROv2 yielding the most stable gains for CoT, and SIMBA providing the largest benefits for ReAct agents, particularly at larger model scales. Behavioral analyses reveal that SIMBA encourages more direct reasoning paths by applying heuristics, thereby improving numerical comparison abilities in CoT reasoning and helping avoid unnecessary tool calls in ReAct agents. Across different prompting techniques, CoT remains effective for tabular fact checking, especially with smaller models. Although ReAct agents built with larger models can achieve competitive performance, they require careful instruction optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。