arXiv:2604.19642cs.CL2026-04

超小模型本地即时生成前4-8词,云端续写实现零延迟响应。

Micro Language Models Enable Instant Responses

论文配图:Micro Language Models Enable Instant Responses
图 1 · 摘自论文原文
  • 8M-30M参数微模型本地运行,首句快速生成。
  • 实测性能媲美70M-256M级主流模型,支持无缝衔接。
  • 适合智能手表等资源受限设备,提升交互实时性。

智能手表、智能眼镜等边缘设备因功耗与算力限制,无法持续运行最小的100M-1B参数语言模型,而云端推理又带来数秒延迟,破坏响应感。本文提出微语言模型(μLMs):参数量仅8M-30M,可在本地瞬间生成上下文相关的前4-8个词,同时由云端模型完成后续生成,从而隐藏云延迟。实验表明,极端小型化下仍可实现有效语言生成,μLMs性能达到现有70M-256M级模型水平。我们设计协同生成框架,将云端模型定位为“续写者”而非“回答者”,实现句子中段无缝交接,并通过三种错误纠正机制保障结构化容错恢复。实证显示,μLMs能发起响应,由大模型无缝续写,证明了数量级差异的不对称协作可行,为极低资源设备解锁了实时人工智能。模型与演示代码见https://github.com/Sensente/micro_language_model_swen_project。

原文摘要 · Abstract (English)

Edge devices such as smartwatches and smart glasses cannot continuously run even the smallest 100M-1B parameter language models due to power and compute constraints, yet cloud inference introduces multi-second latencies that break the illusion of a responsive assistant. We introduce micro language models ($μ$LMs): ultra-compact models (8M-30M parameters) that instantly generate the first 4-8 words of a contextually grounded response on-device, while a cloud model completes it; thus, masking the cloud latency. We show that useful language generation survives at this extreme scale with our models matching several 70M-256M-class existing models. We design a collaborative generation framework that reframes the cloud model as a continuator rather than a respondent, achieving seamless mid-sentence handoffs and structured graceful recovery via three error correction methods when the local opener goes wrong. Empirical results show that $μ$LMs can initiate responses that larger models complete seamlessly, demonstrating that orders-of-magnitude asymmetric collaboration is achievable and unlocking responsive AI for extremely resource-constrained devices. The model checkpoint and demo are available at https://github.com/Sensente/micro_language_model_swen_project.

边缘计算微模型实时响应协同生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。