0.1B小模型实现多模态输入输出,支持语音与文本双向交互。
MiniMind-O Technical Report: An Open Small-Scale Speech-Native Omni Model

- 用中间层语义桥接跨模态信息,提升小模型理解能力。
- 语音克隆相似度达0.5995,端到端生成误差率仅0.0897。
- 开源完整训练数据与代码,适合研究小规模多模态系统者参考。
MiniMind-O 是一个基于 MiniMind 语言模型构建的开源 0.1B 规模多模态模型,可接受文本、语音和图像输入,并输出文本与流式语音。发布内容包括模型代码、检查点及主要 Parquet 格式训练数据集,涵盖文本到音频、图像到文本、音频到音频三类任务,使整个交互流程可直接观测。模型采用完整的 MiniMind 主干作为思考器(Thinker),并独立设计由四个层级构成的说话器(Talker),使用冻结的 SenseVoice-Small 和 SigLIP2 编码器提取语音与图像特征,通过轻量级 MLP 投影器映射后注入模态占位位置。说话器读取中间层思考器状态与自回归八层 Mimi-code 缓冲区。语音控制通过专用说话人标记、右对齐参考编码提示及预计算的 CAM++ 说话人嵌入实现,确保语音条件保留在音频码上下文中而非独立模块。在 768 维说话器下,密集型与 MoE 变体在思考器-说话器一致性评估中平均词错误率分别为 0.0897 与 0.0900,整体语音克隆相似度为 0.5995 与 0.5937。论文不仅呈现了可运行系统,还提炼出三项小规模多模态模型的关键设计:中间层语义桥接、开放的多模态序列格式、参数高效的八码本接口。
原文摘要 · Abstract (English)
MiniMind-O is an open 0.1B-scale omni model built on the MiniMind language model. It accepts text, speech, and image inputs, and returns both text and streaming speech. The release includes model code, checkpoints, and the main Parquet training datasets for text-to-audio, image-to-text, and audio-to-audio training, making the complete interaction loop directly inspectable. The model uses a full MiniMind backbone as the Thinker and an independent four-layer Talker made from MiniMind blocks. Frozen SenseVoice-Small and SigLIP2 encoders provide speech and image features, which are mapped by lightweight MLP projectors and injected at modality-placeholder positions. The Talker reads a middle-layer Thinker state together with an autoregressive eight-layer Mimi-code buffer. Speaker control is handled by a dedicated speaker token, right-aligned reference codec prompts, and precomputed CAM++ speaker embeddings, so voice conditioning remains part of the audio-code context rather than a separate TTS module. With a 768-dimensional Talker, the dense and MoE variants reach average CERs of 0.0897 and 0.0900 in Thinker--Talker consistency evaluation, with overall voice-cloning similarities of 0.5995 and 0.5937. Beyond reporting a working system, the paper identifies three scale-critical design choices for small omni models: middle-layer semantic bridging, a released multimodal sequence format, and a parameter-efficient eight-codebook interface.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。