arXiv:2603.27538cs.CVcs.CL2026-03被引 24

将视觉音频等模态统一为离散令牌,实现跨模态自回归生成。

LongCat-Next: Lexicalizing Modalities as Discrete Tokens

  • 提出DiNA框架,用离散令牌统一表示多模态信息。
  • dNaViT可任意分辨率进行图像离散化,支持端到端建模。
  • 长尾任务表现突破,适合多模态生成与理解场景。

主流的下一令牌预测(NTP)范式通过离散自回归建模推动了大语言模型的发展。然而,当前多模态系统仍以语言为中心,常将非语言模态作为外部附加物,导致架构碎片化且整合不佳。为此,我们提出离散原生自回归(DiNA)框架,将多模态信息统一表示在共享离散空间中,实现跨模态一致的自回归建模。核心创新是离散原生任意分辨率视觉变换器(dNaViT),可在任意分辨率下完成图像的离散化与反离散化,将连续视觉信号转化为分层离散令牌。基于此,我们构建了LongCat-Next,一个原生多模态模型,在单一自回归目标下处理文本、视觉和音频,几乎无需模态特异性设计。作为工业级基础模型,它在单一框架内实现“看、画、说”,在多个多模态基准测试中表现优异。尤其突破了离散视觉建模在理解任务上的性能瓶颈,并提供统一方法有效调和理解与生成间的冲突。为推动原生多模态研究,我们开源LongCat-Next及其分词器。

原文摘要 · Abstract (English)

The prevailing Next-Token Prediction (NTP) paradigm has driven the success of large language models through discrete autoregressive modeling. However, contemporary multimodal systems remain language-centric, often treating non-linguistic modalities as external attachments, leading to fragmented architectures and suboptimal integration. To transcend this limitation, we introduce Discrete Native Autoregressive (DiNA), a unified framework that represents multimodal information within a shared discrete space, enabling a consistent and principled autoregressive modeling across modalities. A key innovation is the Discrete Native Any-resolution Visual Transformer (dNaViT), which performs tokenization and de-tokenization at arbitrary resolutions, transforming continuous visual signals into hierarchical discrete tokens. Building on this foundation, we develop LongCat-Next, a native multimodal model that processes text, vision, and audio under a single autoregressive objective with minimal modality-specific design. As an industrial-strength foundation model, it excels at seeing, painting, and talking within a single framework, achieving strong performance across a wide range of multimodal benchmarks. In particular, LongCat-Next addresses the long-standing performance ceiling of discrete vision modeling on understanding tasks and provides a unified approach to effectively reconcile the conflict between understanding and generation. As an attempt toward native multimodality, we open-source the LongCat-Next and its tokenizers, hoping to foster further research and development in the community. GitHub: https://github.com/meituan-longcat/LongCat-Next

多模态离散建模自回归视觉编码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。