让视觉智能体根据时间预算自动调整推理策略,避免提前中断失败。
AdaTurn: Budget-Aware Test-Time Scaling for Active Visual Perception Agents

- 根据可用回合数动态调整行为,显式训练预算边界响应机制。
- 在4轮限制下,视觉任务准确率从36.7%提升至47.6%。
- 适用于资源受限场景,支持多模型和跨任务迁移。
主动视觉智能体通过多轮推理与图像交互解决细粒度图像任务。然而部署时的推理轮次预算不固定:有些请求允许长轨迹,有些则需在严格轮次限制下行动。现有方法将预算视为隐藏变量训练策略,当实际预算低于模型偏好时,交互常在得出有效答案前被截断,称为“灾难性截断”。为此,我们提出AdaTurn,一种预算感知框架,使智能体基于允许的回合数进行条件决策,并显式训练预算带来的边界行为。核心组件强制回答DAPO(FA-DAPO)将超预算事件从掩码或惩罚失败转化为可训练的最终决策步骤,教会模型在无法继续使用工具时合成部分证据。我们还在训练和推理中随机化预算,并引入负载均衡调度器实现高效运行。AdaTurn显著提升低预算下的准确率,例如在4轮时将VisualProbe-Medium准确率从36.7%提升至47.6%,同时保持大预算下的强扩展性,并有效迁移到多种主干网络和通用多模态基准。
原文摘要 · Abstract (English)
Active visual agents solve fine-grained image tasks by interleaving reasoning with image-grounding actions across multiple turns. However, deployment-time rollout budgets are rarely fixed: some requests permit long rollouts, while others require the agent to act under a tight turn limit. Existing methods train the policy as if the rollout budget were hidden, so when the available budget is smaller than the trajectory the agent prefers, the interaction is often truncated before any valid answer is produced; we term this failure \emph{catastrophic truncation}. To overcome this challenge, we present AdaTurn, a budget-aware framework that conditions the agent on the allowed number of turns and explicitly trains the boundary behavior induced by the budget. Our key component, Forced-Answer DAPO (FA-DAPO), converts the over-budget event from a masked or penalized failure into a trainable final-decision step, teaching the model to synthesize partial evidence when further tool use is no longer possible. We further randomize rollout budgets during both training and inference and introduce a load-balanced scheduler that makes such operations practical. AdaTurn substantially improves low-budget accuracy, for example raising VisualProbe-Medium from 36.7% to 47.6% at four turns, while preserving strong scaling at larger budgets and transferring effectively to multiple backbones and general multimodal benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。