让浏览器代理通过观察用户操作自动学会工作并共享知识。
cotomi Act: Learning to Automate Work by Watching You
- 通过观察用户行为，用压缩历史和动态动作选择实现多步骤任务执行
- 在WebArena评测中达80.4%成功率，超越人类基准78.2%
- 能自动生成任务看板与维基知识，适合需要自动化办公的用户
如果浏览器代理能通过观察你操作来学习工作，会怎样？我们提出cotomi Act，一种基于浏览器的计算机使用代理，兼具可靠的多步任务执行能力与从用户行为中持续积累的组织性知识。执行方面，采用带自适应懒惰观察、基于文本差分的历史压缩、粗粒度动作及测试时通过最佳N选一进行缩放的代理框架，在179个任务的WebArena人工评估子集上达到80.4%准确率，超过报告的人类基准78.2%。在组织知识方面，设计了一条行为到知识的流水线，被动观察用户浏览行为，并逐步抽象为可编辑的任务看板与维基文档，通过共享工作区由用户与代理共同维护。受控代理评估验证了随着行为衍生知识积累，任务成功率随之提升。在真实演示中，参与者在浏览器中直接与系统交互，下达任务并全程观察其自主执行与知识管理过程。
原文摘要 · Abstract (English)
What if a browser agent could learn your work simply by watching you do it? We present cotomi Act, a browser-based computer-using agent that combines reliable multi-step task execution with persistent organizational knowledge learned from user behavior. For execution, an agent scaffold with adaptive lazy observation, verbal-diff-based history compression, coarse-grained actions, and test-time scaling via best-of-N action selection achieves 80.4% on the 179-task WebArena human-evaluation subset, exceeding the reported 78.2% human baseline. For organizational knowledge, a behavior-to-knowledge pipeline passively observes the user's browsing and progressively abstracts it into artifacts (task boards, wiki) exposed through a shared workspace editable by both user and agent. A controlled proxy evaluation confirms that task success improves as behavior-derived knowledge accumulates. In our live demonstration, attendees interact with the system in a real browser, issuing tasks and observing end-to-end autonomous execution and shared knowledge management.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。