arXiv:2606.30294cs.AIcs.HC2026-06

让AI自动演示范例并实时回答语音问题,还能自我修复错误。

Rehearsed Multi-Agent Live Product Demonstrations with Real-Time Voice Question Answering

  • 用多智能体系统结合界面与代码分析,生成可交互的演示脚本。
  • 在4个真实应用上测试,定位成功率最高达92%,修复后达100%。
  • 适合需要高频演示的软件团队,尤其看重自动化与容错能力。

现场产品演示是软件组织中重复性高、成本高的活动:需人工选择功能,在运行中的应用上执行操作,连贯讲述,并实时回答问题。现有自动化仅覆盖片段——通用浏览器代理聚焦指令驱动的任务完成,演示视频工具生成固定MP4文件,无法问答且易因界面变化失效。我们提出Rhetor,一个以运行中的网页应用及其源码库为输入的多智能体系统,可生成预排练的实时演示,实现分段同步语音叙述与实时语音问答。其架构创新包括:融合界面探索与代码分析的跨模态特征表示,聚焦层级标记;受界面元素约束的脚本生成器,通过多策略语义定位器调度;预演示排练循环,具备显式收敛与降级至纯叙述的能力;以及运行时同步不变量,将每个浏览器操作与对应叙述段的音频结束事件对齐。在四个部署应用的六次流程中,内部定位器触发率(sigma-bar)在147个脚本动作中为0.31-1.00;在53个动作、完整层级区分的复杂任务中,sigma-bar约为0.92;在公开白板应用Excalidraw上,定位修复步骤使迭代2时收敛至sigma-bar=1.00。我们还定义了涵盖六类应用的十项指标基准协议,以验证设计选择的有效性。

原文摘要 · Abstract (English)

Live product demonstrations are a recurring, high-cost activity in software organizations: a human presenter must select features, dispatch the corresponding interactions on a running application, narrate them coherently, and answer questions in real time. Existing automation addresses only fragments -- generalist browser agents target instruction-conditioned task completion, and demo-video tools produce fixed MP4 artifacts that cannot be questioned and silently break under interface drift. We propose Rhetor, a multi-agent system that takes a running web application and its source-code repository as input and produces a rehearsed live demonstration with segment-synchronized narration and real-time voice question answering. The architectural contributions are a cross-modal feature representation that merges UI exploration with source-code analysis into features tagged with discrete focus tiers, a grounded scripter constrained to UI elements observed during exploration and dispatched through multi-strategy semantic locators, a pre-presentation rehearsal loop with explicit convergence and graceful degradation to narration-only segments, and a runtime synchronization invariant that ties each browser action to the audio-end event of its narration segment. Across six pipeline sessions on four deployed applications -- including the public-domain whiteboard application Excalidraw -- the rehearser's internal locator-firing rate (sigma-bar) spans 0.31-1.00 over 147 scripted actions; on the substantial workload (53 actions, full tier differentiation), sigma-bar is approximately 0.92, and on the public-domain reference point the locator-repair step drives convergence to sigma-bar = 1.00 at iteration 2. We additionally define a benchmark protocol of ten metrics across six application categories that would establish, beyond the case study, whether each design choice contributes positively.

多智能体产品演示语音问答自动化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。