通过迭代选例提升大模型代数推理能力，效果优于传统提示。
Iterative In-Context Learning to Enhance LLMs Abstract Reasoning: The Case-Study of Algebraic Tasks
- 用迭代方式挑选少数示例，逐步优化提示效果。
- 简单示例比复杂示例更能提升模型外推性能。
- 适合需要强逻辑推理的数学任务研究者参考。
大语言模型在系统化泛化方面存在挑战，尤其在需要组合规则和处理分布外样本的推理任务中表现不佳。为此，我们提出一种上下文学习方法，通过迭代示例选择策略，逐步构建针对特定任务定制的少样本示例集，以增强通用大模型的泛化能力。以非标准简化规则下的代数表达式求解为例，该规则改变了加法与乘法的优先级。实验表明，大模型在这些数学任务中表现有限。进一步验证发现，结合显式推理指令的迭代示例提示策略能显著提升模型推理能力。关键的是，部分模型在使用简单示例而非符合测试数据分布的复杂示例时，展现出更优的泛化性能。
原文摘要 · Abstract (English)
LLMs face significant challenges in systematic generalization, particularly when dealing with reasoning tasks requiring compositional rules and handling out-of-distribution examples. To address these challenges, we introduce an in-context learning methodology that improves the generalization capabilities of general purpose LLMs. Our approach employs an iterative example selection strategy, which incrementally constructs a tailored set of few-shot examples optimized to enhance model's performance on a given task. As a proof of concept, we apply this methodology to the resolution of algebraic expressions involving non-standard simplification rules, according to which the priority of addition and multiplication is changed. Our findings indicate that LLMs exhibit limited proficiency in these mathematical tasks. We further demonstrate that LLMs reasoning benefits from our iterative shot selection prompting strategy integrated with explicit reasoning instructions. Crucially, our experiments reveal that some LLMs achieve better generalization performances when prompted with simpler few-shot examples rather than complex ones following the test data distribution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。