本地语音助手AnovaX用LLM规划任务,实现无键盘交互的跨应用操作。
AnovaX: A Local, Multi-Agent Voice Assistant with LLM Planning, Typed Executors, and Adaptive Recovery
- 基于LLM生成计划,通过多智能体架构执行类型化操作
- 支持失败自恢复与并发任务协调,可远程手机控制
- 全程本地运行,不依赖云端,保障隐私且延迟低
桌面语音助手仍依赖云端处理,仅提供固定技能。我们提出AnovaX,一个运行在用户本地的小型首任助手,将桌面本身作为操作界面。单个Python进程整合唤醒词检测、语音处理、使用Gemini的LLM规划器(输出JSON计划)、白名单/黑名单安全层、多智能体调度器(在有限线程池中生成类型化子智能体),以及核心步骤失败时触发的自适应恢复循环。每个工具对应专用智能体类(如AppAgent、TypingAgent、BrowserAgent等),具备独立超时、重试策略和共享资源锁。递归MetaAgent允许规划器最多两层嵌套地委派子目标。恢复循环采用紧凑的ReAct式提示,并通过预执行只读工具隐藏Gemini延迟。配套Flask服务器通过本地WiFi提供手机友好的远程控制,实时同步所有智能体生命周期事件,并以MJPG流回传屏幕画面,让用户观察远程命令执行过程。项目重点并非挑战Siri或Alexa,而是证明一个几千行代码的可读系统足以完成开应用、输入内容、搜索、协调并发动作、从单步失败中恢复,并完全由另一房间的手机驱动——而无需大模型触碰键盘。
原文摘要 · Abstract (English)
Desktop voice assistants are still dominated by cloud pipelines that ship raw audio off the machine and expose a fixed set of skills. We describe AnovaX, a small local-first assistant that runs entirely on the user's computer and treats the desktop itself as its action surface. A single Python process wires together a wake-word gate, a speech pipeline, an LLM planner (Gemini) that emits a JSON plan of tool calls, a whitelist-and-denylist safety layer, a multi-agent orchestrator that translates each plan into typed child agents on a bounded thread pool, and an adaptive recovery loop that takes over whenever a core step fails. Every tool corresponds to a specialized agent class (AppAgent, TypingAgent, BrowserAgent and six others) with its own timeout, retry policy, and shared-resource locks. A recursive MetaAgent lets the planner delegate a sub-goal back to itself, capped at two levels of nesting. The recovery loop uses a compact ReAct-style prompt and hides Gemini's latency behind speculative execution of read-only tools. A companion Flask server exposes a phone-friendly remote over the local WiFi, mirrors every agent lifecycle event to the phone in real time, and streams the laptop's screen back over MJPEG so the user can watch remote commands land as they run. The point of the project is less to compete with Siri or Alexa than to show that a legible, few-thousand-line assistant is enough to open apps, type into them, run searches, coordinate concurrent actions, recover from single-step failures, and be driven entirely from a phone in another room -- without the LLM ever touching the keyboard.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。