Gander实现多模态实时交互与智能代理一体化,支持自由打断和主动反馈。
Omni Interaction Agent Technical Report

- 采用脑-小脑协同架构,分别处理实时交互与复杂推理任务。
- 通过流式思维-说话者结构实现低延迟连续对话,支持任意时刻中断。
- 在真实场景中表现稳健,适合需要自然交互的智能助手应用。
本文提出Gander,一个统一多模态感知、实时交互与智能体能力的端到端模型。不同于传统的轮次式交互范式,Gander持续接收视频、语音、文本等多模态流式输入,支持日常对话与复杂工作流场景下的自然全双工交互。用户可随时打断,模型亦可主动提供中间反馈或追问。为原生支持此能力,Gander采用两项核心设计:1)脑-小脑协同框架,其中小脑负责实时交互与多轮对话,大脑处理复杂推理与高层智能体任务,二者通过工具调用与智能体编排运行时持续交互;2)小脑基于流式思维-说话者架构,将用户输入与模型输出在块级别扁平化为有序令牌流,实现低延迟连续交互的统一表示。我们在四个维度对Gander进行全面评估:对话能力、多模态理解、交互能力与智能体智能。内部人工评估显示,Gander在保持当前开源领先模型自然表达口语对话能力的同时,实现了具有竞争力的多模态交互性能。其在背景噪声干扰、多方交互及附带反馈等挑战性真实场景中也表现出良好鲁棒性。我们已公开Gander及其模型、代码与数据,以促进社区进一步研究与发展。
原文摘要 · Abstract (English)
In this work, we present Gander, an end-to-end model that unifies omni perception, realtime interaction, and agentic capabilities within a single framework. In contrast to turn-based conventional paradigms, Gander continuously receives streaming inputs across multiple modalities, including video, speech, and text, enabling natural full-duplex interaction in both everyday conversations and complex workflow-oriented agent scenarios. Users can interrupt the model at any time, while the model can also proactively provide intermediate feedback or ask follow up questions. To natively support these capabilities, Gander adopts two key architectural designs: 1) It employs a Cerebellum-Brain collaborative framework, in which the Cerebellum is responsible for realtime interaction and omni conversational capabilities, while the Brain handles complex reasoning and higher-level agentic tasks. The two components interact continuously through tool calling and the agent orchestration runtime. 2) The Cerebellum is built upon a streaming Thinker-Talker architecture, user inputs and model outputs are further flattened into an ordered token stream at the chunk level, providing a unified representation for low latency, continuous interaction. We conduct comprehensive evaluations of Gander across four dimensions: conversational ability, omni understanding, interactive capability, and agentic intelligence. Internal human evaluations demonstrate that Gander maintains the natural and expressive spoken dialogue capabilities of SOTA open source models while achieving competitive performance in omni interaction. Gander also demonstrates robustness in challenging real-world scenarios, including background noise interference, multi-party interactions, and backchannel communication. We release Gander together with its models, code, and data to facilitate further research and development in the community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。