用意图感知的评分模型提升电脑操作智能体的动作质量
IntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agents
- 通过意图编码器区分相似动作的不同决策理由
- 在未见环境中使任务成功率提升6.9个百分点
- 适用于需要高可靠性桌面操作的智能体系统
计算机使用智能体（CUAs）利用大语言模型在桌面环境中执行图形界面操作，但缺乏对动作质量的评估，导致不可逆错误并引发后续连锁失效。我们提出 IntentScore，一种基于计划的奖励模型，从跨越三个操作系统的398,000条离线GUI交互步骤中学习评分。该模型采用两种互补目标：对比对齐以衡量状态-动作相关性，边际排序以评估动作正确性。架构上，将候选动作的规划意图嵌入动作编码器，实现对具有相似行为但不同推理逻辑的动作的区分。IntentScore在保留测试集上达到97.5%的成对区分准确率。将其部署为Agent S3在OSWorld环境中的重排序器（训练中未见过该环境），任务成功率提升6.9分，表明从异构离线轨迹中学到的奖励估计可泛化至未见过的智能体与任务分布。
原文摘要 · Abstract (English)
Computer-Use Agents (CUAs) leverage large language models to execute GUI operations on desktop environments, yet they generate actions without evaluating action quality, leading to irreversible errors that cascade through subsequent steps. We propose IntentScore, a plan-aware reward model that learns to score candidate actions from 398K offline GUI interaction steps spanning three operating systems. IntentScore trains with two complementary objectives: contrastive alignment for state-action relevance and margin ranking for action correctness. Architecturally, it embeds each candidate's planning intent in the action encoder, enabling discrimination between candidates with similar actions but different rationales. IntentScore achieves 97.5% pairwise discrimination accuracy on held-out evaluation. Deployed as a re-ranker for Agent S3 on OSWorld, an environment entirely unseen during training, IntentScore improves task success rate by 6.9 points, demonstrating that reward estimation learned from heterogeneous offline trajectories generalizes to unseen agents and task distributions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。