让大模型通过自我反思迭代优化答案，理论可证且推理成本低。
Provable and Practical In-Context Policy Optimization for Self-Improvement
- 用上下文内策略优化，不改参数只凭自评奖励迭代改进回答。
- 在数学推理任务中表现顶尖，推理开销远低于同类方法。
- 基于最小熵选择提升自评奖励可靠性，适合需要高精度的推理场景。
我们研究测试时扩展，即模型在推理阶段通过多轮自我反思改进答案。提出上下文内策略优化（ICPO），代理在不修改参数的前提下，利用自评或外部观测奖励在上下文中优化响应。理论上证明，在新型费雪加权对数匹配目标下充分预训练后，单层线性自注意力模型可严格模仿线性老虎机的策略优化算法。基于此理论，提出最小熵ICPO（ME-ICPO），一种实用算法，通过迭代使用响应与自评奖励在推理时上下文内精炼答案。通过选择最低熵的响应及其奖励，ME-ICPO借助多数投票增强自评奖励的鲁棒性。在标准数学推理任务中，ME-ICPO达到竞争性、顶级性能，同时推理成本远低于其他推理时算法。总体而言，ICPO为大模型的自我反思提供了原则性理解，并为数学推理的测试时扩展带来实际效益。
原文摘要 · Abstract (English)
We study test-time scaling, where a model improves its answer through multi-round self-reflection at inference. We introduce In-Context Policy Optimization (ICPO), in which an agent optimizes its response in context using self-assessed or externally observed rewards without modifying its parameters. To explain this ICPO process, we theoretically show that with sufficient pretraining under a novel Fisher-weighted logit-matching objective, a single-layer linear self-attention model can provably imitate policy-optimization algorithm for linear bandits. Building on this theory, we propose Minimum-Entropy ICPO (ME-ICPO), a practical algorithm that iteratively uses its response and self-assessed reward to refine its response in-context at inference time. By selecting the responses and their rewards with minimum entropy, ME-ICPO ensures the robustness of the self-assessed rewards via majority voting. Across standard mathematical reasoning tasks, ME-ICPO attains competitive, top-tier performance while keeping inference costs affordable compared with other inference-time algorithms. Overall, ICPO provides a principled understanding of self-reflection in LLMs and yields practical benefits for test-time scaling for mathematical reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。