让推理模型在运行时自我修正,通过中间步骤验证提升准确率。
interwhen: A Generalizable Framework for Steering Reasoning Models with Test-time Verification
- 通过周期性监控推理过程,动态触发验证器检查中间结果。
- 在数学与逻辑任务上实现接近完美的准确率,仅用少量token。
- 可自动从自然语言政策生成验证代码,适合无调优的智能体使用。
推理模型产生长序列的中间决策和工具调用,测试时验证对确保正确性至关重要。现有方法或仅验证最终答案(忽略早期错误),或依赖分支验证策略探索多条路径。本文提出interwhen,一种单轨迹验证框架,通过向推理过程提供中间反馈来引导模型行为。针对两个关键挑战:首先,传统方法需提示工程或固定步骤分解以获取可验证状态;interwhen设计监控系统,周期性轮询推理轨迹并分叉推理以恢复中间状态,验证器异步运行,正确执行时开销极小,仅在异常时介入。其次,除数学与代码外,过程验证面临验证器稀缺问题;interwhen通过从自然语言政策文档自动合成验证器解决,可生成基于代码的验证器,包括在Lean和z3中可证明正确的验证器。两项贡献共同构成即插即用的测试时验证系统,显著提升任意推理代理的任务完成率与策略合规性。在编码数学或逻辑约束的推理基准上,interwhen使推理模型达到近似完美准确率,仅需基线分数之一的token量。在基于策略的验证器生成的智能体基准上,无需微调即可提升任务质量,例如Qwen3-30B在tau2-bench电信领域任务完成率从32%升至87%。代码已开源。
原文摘要 · Abstract (English)
Reasoning models produce long traces of intermediate decisions and tool calls, making test-time verification important for ensuring correctness. Existing approaches either verify only the final answer, which misses early errors, or rely on branch-and-verify strategies that explore multiple trajectories. We introduce interwhen, a single-trajectory verification framework that steers model behavior by providing feedback on intermediate reasoning traces. It addresses two key challenges. First, given a set of verifiers, obtaining verifiable states from the reasoning trace typically requires prompt engineering or external task decomposition into fixed steps. Instead, we propose a monitoring system that periodically polls the reasoning trace and forks inference of the reasoning model to recover intermediate states. Verifiers are run asynchronously alongside generation, adding negligible overhead on correct executions and intervening only when violations occur. Second, beyond math and code, a central challenge for process verification is the scarcity of verifiers. interwhen addresses this through automatic verifier synthesis from natural-language policy documents. Given a policy, it can generate code-based verifiers, including provably correct verifiers in Lean and z3. Together, these contributions yield a plug-and-play test-time verification system that can improve task completion and policy compliance of any reasoning agent. On reasoning benchmarks where policies encode mathematical or logical constraints, interwhen achieves near-perfect accuracy for reasoning models using a fraction of the tokens of baselines. On agentic benchmarks with policy-based verifier generation, it enables improvements in task quality for SLMs without any finetuning, e.g., task completion rate of Qwen3-30B jumps from 32% to 87% on the telecom domain in tau2-bench. Code at https://github.com/microsoft/interwhen.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。