根据实例自适应调整提示,提升大模型零样本推理效果。
Instance-adaptive Zero-shot Chain-of-Thought Prompting
- 基于信息流动分析,动态区分好坏提示。
- 在GSM8K等任务上实现一致性能提升。
- 适合追求高效零样本推理的开发者和研究者。
零样本思维链(CoT)提示已成为提升大语言模型在现实推理任务中表现的一种简单有效策略。然而,对所有实例统一使用单一任务级提示的局限性在于:一个提示无法适配所有情况。本文提出一种实例自适应提示算法(IAP),通过自适应区分优质与劣质提示,改进零样本CoT推理。我们从信息流角度分析大模型,发现问题到提示、问题到推理路径的信息交互共同影响推理结果。高质量的推理需要提示获取问题语义信息,推理过程则需直接从问题和间接通过提示获得充分信息。缺乏任一环节均可能导致效果下降。实验在LLaMA-2、LLaMA-3和Qwen上进行,覆盖数学、逻辑与常识推理任务(如GSM8K、MMLU、Causal Judgement),结果一致优于其他采用人工调优提示或复杂流程的方法,验证了该机制的重要性。
原文摘要 · Abstract (English)
Zero-shot Chain-of-Thought (CoT) prompting emerges as a simple and effective strategy for enhancing the performance of large language models (LLMs) in real-world reasoning tasks. Nonetheless, the efficacy of a singular, task-level prompt uniformly applied across the whole of instances is inherently limited since one prompt cannot be a good partner for all, a more appropriate approach should consider the interaction between the prompt and each instance meticulously. This work introduces an instance-adaptive prompting algorithm as an alternative zero-shot CoT reasoning scheme by adaptively differentiating good and bad prompts. Concretely, we first employ analysis on LLMs through the lens of information flow to detect the mechanism under zero-shot CoT reasoning, in which we discover that information flows from question to prompt and question to rationale jointly influence the reasoning results most. We notice that a better zero-shot CoT reasoning needs the prompt to obtain semantic information from the question then the rationale aggregates sufficient information from the question directly and via the prompt indirectly. On the contrary, lacking any of those would probably lead to a bad one. Stem from that, we further propose an instance-adaptive prompting strategy (IAP) for zero-shot CoT reasoning. Experiments conducted with LLaMA-2, LLaMA-3, and Qwen on math, logic, and commonsense reasoning tasks (e.g., GSM8K, MMLU, Causal Judgement) obtain consistent improvement, demonstrating that the instance-adaptive zero-shot CoT prompting performs better than other task-level methods with some curated prompts or sophisticated procedures, showing the significance of our findings in the zero-shot CoT reasoning mechanism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。