arXiv:2512.16963cs.LG2025-12被引 1

用重建误差当路由信号,让模型自动选专家模块。

Compression is Routing: Reconstruction Error as an Intrinsic Signal for Modular Language Models

  • 用自编码器压缩序列,重建误差作为内在路由依据。
  • 压缩比达64倍,代码域重建准确率99.47%,随机序列仅0.57%。
  • 无需显式门控网络,适合长文本处理与持续学习场景。

当前大语言模型面临上下文长度限制、推理成本高和持续学习中的灾难性遗忘三大挑战。虽然混合专家(MoE)架构缓解部分矛盾,但其路由机制通常依赖显式训练的辅助分类器,增加系统复杂度且对跨领域输入缺乏可解释性。基于‘压缩即智能’的假设,本文提出新架构理念:压缩即路由。我们训练了一个8700万参数的端到端Transformer自编码器,实现64倍序列压缩(将512个标记压缩为8个隐向量)。实验表明,该压缩器具备极强的领域判别能力:在领域内(代码)验证集上重建准确率达99.47%;在半域外(维基文本)降至47.76%;完全域外(随机序列)进一步暴跌至0.57%。这种系统性性能差异确立了重建误差作为内在分布指纹的有效性。基于此,我们提出直接利用重建残差自动调度专家模块,无需显式门控网络。该机制具备良好可扩展性,并为超长上下文的‘显存压缩’提供了新视角。本报告旨在验证该基础架构的物理有效性,为下一代可扩展模块化神经网络提供新思路。

原文摘要 · Abstract (English)

Current Large Language Models (LLMs) face three major challenges: context length limitations, high inference costs, and catastrophic forgetting during continual learning. While Mixture-of-Experts (MoE) architectures mitigate some of these conflicts, their routing mechanisms typically rely on explicitly trained auxiliary classifiers. This not only increases system complexity but also often lacks interpretability when handling mixed-domain inputs. Building upon the premise that ``Compression is Intelligence,'' this paper proposes a novel architectural philosophy: Compression is Routing. We trained an 87M-parameter end-to-end Transformer Autoencoder, achieving a 64x sequence length compression (compressing 512 tokens into 8 latent vectors). Experimental results demonstrate that this compressor possesses extreme domain discriminative capability: it achieves a reconstruction accuracy of 99.47% on the in-domain (code) validation set; accuracy drops sharply to 47.76% on a semi-out-of-distribution domain (Wiki text); and further plummets to just 0.57% on a fully out-of-distribution domain (random sequences). This extreme and systematic performance discrepancy establishes the validity of reconstruction error as an Intrinsic Distribution Fingerprint. Based on this, we propose that expert modules can be automatically scheduled using reconstruction residuals directly, without the need for explicit gating networks. This mechanism offers excellent scalability. Furthermore, this architecture provides a new perspective on ``VRAM compression'' for handling ultra-long contexts. This report aims to verify the physical validity of this foundational architecture, offering a new research perspective for the next generation of scalable modular neural networks.

模块化模型压缩路由自编码器持续学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。