提出无需分词器的连续球面蒸馏方法，让模型直接处理字节序列。
HoloByte: Continuous Hyperspherical Distillation for Tokenizer-Free Modeling
- 用连续球面空间替代离散分词，通过可逆旋转压缩字节序列
- 将注意力复杂度从O(N²D)降至O(N²/W²D + ND²)，显著提速
- 适合追求高效、无词汇依赖的序列建模研究者
序列建模通常依赖离散子词分词以克服原生字节级注意力的O(N²)计算不可行性。然而，这种启发式量化引入了人为的形态边界，依赖词表，并破坏优化空间的连续性。为解决这一矛盾，我们提出HoloByte：一种严格无分词器的框架，采用连续球面蒸馏。HoloByte将离散字节序列划分为固定容量块，并通过可逆、保持维度的正交旋转算子投影到连续且严格有界的球面流形上。该空间叠加使宏观Transformer仅在压缩的连续表示上操作，形式上将精确注意力时间复杂度从O(N²D)降低至O(N²/W²D + ND²)。随后的局部因果微解码器还原这些表示以计算精确字节级分布。为控制此连续轨迹，我们提出双目标公式，包含数学精确的全息潜在均方误差，严格限制梯度并保证渐近稳定性。理论上，我们推导出最小嵌入维度D = Ω(W ln |V|)，以确保从连续流形中无误差恢复离散表示。实验上，在严格匹配参数约束下，HoloByte系统性优于可比的离散字节对编码（BPE）基线。结果确立连续球面蒸馏作为无词表序列建模的数学严谨且计算可行的基础。代码见https://github.com/VladimerKhasia/HoloByte。
原文摘要 · Abstract (English)
Sequence modeling universally relies on discrete subword tokenization to circumvent the $\mathcal{O}(N^2)$ computational intractability of native byte-level attention. However, this heuristic quantization imposes artificial morphological boundaries, enforces vocabulary dependence, and fractures the continuity of the optimization landscape. To resolve this dichotomy, we introduce \textbf{HoloByte}: a strictly tokenizer-free framework utilizing Continuous Hyperspherical Distillation. HoloByte partitions discrete byte sequences into fixed-capacity chunks and projects them into a continuous, strictly bounded hyperspherical manifold via an invertible, dimension-preserving orthogonal rotation operator. This spatial superposition allows a macroscopic transformer to operate exclusively on compressed continuous representations, formally reducing the exact attention time complexity from $\mathcal{O}(N^2D)$ to $\mathcal{O}\left( \frac{N^2}{W^2}D + ND^2 \right)$. A localized causal micro-decoder subsequently unbinds these representations to compute exact byte-level distributions. To govern this continuous trajectory, we propose a dual-objective formulation incorporating a mathematically precise Holographic Latent Mean Squared Error, which strictly bounds the gradient and guarantees asymptotic stability. Theoretically, we derive the minimal embedding dimension $D = Ω(W \ln |\mathcal{V}|)$ required to ensure error-free discrete recovery from the continuous manifold. Empirically, under strictly matched parameter constraints, HoloByte is systematically outperforming a comparable discrete Byte-Pair Encoding (BPE) baseline. These results establish Continuous Hyperspherical Distillation as a mathematically rigorous and computationally tractable foundation for vocabulary-invariant sequence modeling. The code is available at https://github.com/VladimerKhasia/HoloByte
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。