用分层强化学习实现轻量级桌面自动化，效率远超大模型。
Towards General Computer Control with Hierarchical Agents and Multi-Level Action Spaces
- 分两层决策：管理器统筹任务，子策略执行具体操作。
- 简单任务成功率92.1%，复杂任务58.8%，推理速度翻倍提升。
- 模型仅1500万参数，适合手机等设备实时运行。
通过软件控制桌面应用仍是基础但未被充分解决的问题。现有多模态大语言模型（MLLM）虽能根据截图和任务指令生成键盘与鼠标事件，但存在推理延迟高、长时程稀疏奖励任务样本效率低、难以在设备端部署等问题。本文提出轻量级分层强化学习框架ComputerAgent，将操作系统控制建模为两级选项过程（管理器与子策略），采用三模态状态编码器（截图、任务ID、数值状态）应对视觉与上下文多样性，引入元动作与早停机制减少无效交互，并使用紧凑视觉骨干加小型策略网络实现设备端推理（1500万参数）。在135个真实桌面任务上，ComputerAgent在简单任务（<8步）中取得92.1%成功率，在复杂任务（≥8步）中达58.8%，在简单场景下性能媲美或超越2000亿参数的MLLM基线，模型规模缩小超四数量级，推理时间减半。结果表明，分层强化学习为计算机控制提供了一种高效可扩展的替代方案。
原文摘要 · Abstract (English)
Controlling desktop applications via software remains a fundamental yet under-served problem. Existing multi-modal large language models (MLLMs) ingest screenshots and task instructions to generate keystrokes and mouse events, but they suffer from prohibitive inference latency, poor sample efficiency on long-horizon sparse-reward tasks, and infeasible on-device deployment. We introduce a lightweight hierarchical reinforcement learning framework, ComputerAgent, that formulates OS control as a two-level option process (manager and subpolicy), employs a triple-modal state encoder (screenshot, task ID, numeric state) to handle visual and contextual diversity, integrates meta-actions with an early-stop mechanism to reduce wasted interactions, and uses a compact vision backbone plus small policy networks for on-device inference (15M parameters). On a suite of 135 real-world desktop tasks, ComputerAgent attains 92.1% success on simple tasks (<8 steps) and 58.8% on hard tasks (>=8 steps), matching or exceeding 200B-parameter MLLM baselines on simple scenarios while reducing model size by over four orders of magnitude and halving inference time. These results demonstrate that hierarchical RL offers a practical, scalable alternative to monolithic MLLM-based automation for computer control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。