动态调度多技能,让机器人更准更快判断物体可操作位置。
Affordance Agent Harness: Verification-Gated Skill Orchestration

- 用路由器自适应选择和调整技能,按需调用
- 验证器通过三重检查确保决策可靠,减少错误提交
- 能复用旧经验,适合复杂开放场景下的交互任务
可操作性定位需在开放世界场景中识别代理应互动的位置与方式,但可操作区域常小、被遮挡、反光或视觉模糊。现有系统虽融合检测、分割、交互想象等多技能,但多采用固定流程,难以匹配实例难度,缺乏对中间错误的精准修复,也无法复用重复物体的经验。这暴露了系统性问题:测试时需获取合适证据,判断证据是否足够可信以做出决策,且在无标签、推理成本受限条件下完成。我们提出Affordance Agent Harness,一个闭环运行时框架,统一异构技能,建立证据库与成本控制机制,利用情景记忆为重复类别提供先验,通过路由器动态选择并参数化技能。专属验证器基于自洽性、跨尺度稳定性与证据充分性进行门控,触发针对性重试;最终裁判融合累积证据与轨迹生成预测。在多个可操作性基准及可控难度子集上实验显示,该方法优于固定流水线基线,在准确率-成本权衡曲线上表现更优,提升定位质量的同时降低平均技能调用次数与延迟。
原文摘要 · Abstract (English)
Affordance grounding requires identifying where and how an agent should interact in open-world scenes, where actionable regions are often small, occluded, reflective, and visually ambiguous. Recent systems therefore combine multiple skills (e.g., detection, segmentation, interaction-imagination), yet most orchestrate them with fixed pipelines that are poorly matched to per-instance difficulty, offer limited targeted recovery from intermediate errors, and fail to reuse experience from recurring objects. These failures expose a systems problem: test-time grounding must acquire the right evidence, decide whether that evidence is reliable enough to commit, and do so under bounded inference cost without access to labels. We propose Affordance Agent Harness, a closed-loop runtime that unifies heterogeneous skills with an evidence store and cost control, retrieves episodic memories to provide priors for recurring categories, and employs a Router to adaptively select and parameterize skills. An affordance-specific Verifier then gates commitments using self-consistency, cross-scale stability, and evidence sufficiency, triggering targeted retries before a final judge fuses accumulated evidence and trajectories into the prediction. Experiments on multiple affordance benchmarks and difficulty-controlled subsets show a stronger accuracy-cost Pareto frontier than fixed-pipeline baselines, improving grounding quality while reducing average skill calls and latency. Project page: https://tenplusgood.github.io/a-harness-page/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。