arXiv:2605.28629cs.CL2026-05中稿 · TASLP

让手机操作智能体学会判断何时该自己做、何时该问人,避免瞎操作也不乱打扰。

Mobile-Aptus: Confidence-Driven Proactive and Robust Interaction in MLLM-based Mobile-Using Agents

论文配图:Mobile-Aptus: Confidence-Driven Proactive and Robust Interaction in MLLM-based Mobile-Using Agents
图 1 · 摘自论文原文
  • 根据置信度主动决策是否执行或求助,减少盲目操作和过度求人。
  • 在4个主流基准上任务成功率平均提升17%,真实场景中成功率达基线26%以上。
  • 适合需要高可靠性、低人工干预的自动化手机操作应用。

多模态大模型使手机操作智能体能自主执行用户指令,但完全自动化常导致无法完成任务仍强行执行(过执行)。已有研究通过训练交互式智能体在无法完成时请求人类帮助来缓解此问题,但这些方法又容易出现过度求助行为。为此,本文提出通用置信度融合框架Mobile-Aptus,实现基于置信度的主动且稳健的交互。该框架分两阶段:第一阶段通过监督微调使智能体输出动作与置信度评分;第二阶段结合语义相似性检索与直接偏好优化,纠正置信度偏差。实验表明,Mobile-Aptus在OS-Kairos、AITZ、Meta-GUI和AndroidControl四个主流基准上均达最优表现,离线测试平均任务成功率提升超过17%;真实动态环境中任务成功率超越基线26%,每条指令仅需0.64次干预。代码已开源。

原文摘要 · Abstract (English)

Recent advancements in multimodal large language models (MLLMs) have shown exceptional potential in enabling mobile-using agents to autonomously execute human instructions. However, fully automated agents often try to execute tasks even when they are unable to resolve them, leading to the problem of over-execution. Previous studies solve it by training a interactive mobile-using agents to let agents request human interaction when agents can not complete user instructions. However, we find that these interactive agents tend to exhibit over-soliciting behavior, relying excessively on human intervention. To mitigate both over-execution and over-soliciting, we propose a universal confidence integration framework that enables confidence-driven proactive and robust interaction in MLLM-based mobile-using agents. The framework consists of two stages: interaction capability empowerment and confidence bias correction. In the interaction capability empowerment stage, agents learn through supervised fine-tuning to output both actions and confidence scores. In the confidence bias correction stage, agents learn to output more accurate confidence scores by combining semantic similarity retrieval with direct preference optimization. Experimental results show Mobile-Aptus achieves state-of-the-art performance on the four popular mobile-using agent benchmarks: OS-Kairos, AITZ, Meta-GUI, and AndroidControl. Mobile-Aptus consistently outperforms all baselines in offline benchmarks, with an average improvement over 17\% in task success rate. In real-world dynamic experiments, Mobile-Aptus surpasses the baseline by 26% in task success rate with only 0.64 intervention steps per instruction. The codes are available at https://github.com/Wuzheng02/Mobile-Aptus.

多模态大模型手机自动化置信度控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。