Syll让个人智能代理跨界面操作，支持用户演示教学与透明审计。
Syll: Open-Source Personal Automation with Cross-Surface Execution
- 通过双向交互层，用户可直接演示操作，系统自动生成可复用技能。
- 支持多模态证据输出（日志、关键帧、确认点），便于审查和控制。
- 所有技能与记忆本地化可编辑，适合希望自定义自动化的人群。
个人AI代理需在API、命令行、网页和桌面图形界面间协同工作，但现有系统多局限于单一接口，且缺乏用户教学与可审计性支持。我们提出Syll，一个开源、自托管的多模态代理运行时，整合MCP/API工具、命令行执行与视觉GUI控制，在模块化架构下实现跨异构界面的协作自动化。核心是双向人机交互层：用户通过直接示范教授流程，Syll将其编译为可重用技能；代理执行过程反向生成多模态证据（日志、关键帧、审批检查点），供审查与干预。Syll还将记忆、技能、流程与治理规则外化为可编辑的本地文件，支持透明检查、扩展与下游开发。实现已验证于Adobe Photoshop、Adobe Audition、Stardew Valley、macOS Finder等生产级应用。机制研究验证了多模态路由、可教学的GUI回放及持久本地化实体的有效性。我们希望Syll能成为用户可教学、可审计、可持续扩展的个人自动化开源基础。
原文摘要 · Abstract (English)
Personal AI agents must increasingly operate across APIs, shells, web surfaces, and desktop GUIs, yet many systems remain tuned to a single interface and offer limited support for user teaching and auditability. We present Syll, an open-source, self-hosted multimodal agent harness that unifies MCP/API tools, CLI execution, and visual GUI control in a modular runtime, enabling agents to coordinate computer use across heterogeneous interfaces while streamlining how users and agents exchange information. At the core of Syll is a bidirectional user-agent interaction layer: users teach procedures through direct demonstration, which Syll compiles into reusable skills; agent execution is translated back into multimodal evidence -- logs, keyframes, and approval checkpoints -- for inspection and control. Syll further externalizes memory, skills, routines, and governance as editable local artifacts, supporting straightforward inspection, extension, and downstream development. Our implementation has been validated on production desktop applications including Adobe Photoshop, Adobe Audition, Stardew Valley, macOS Finder and others. We report mechanism-oriented studies that validate multimodal routing, teachable GUI replay, and persistent local artifacts. We hope Syll can serve as a practical open-source foundation for personal automation that users can teach, inspect, and continuously extend.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。