通过动态插入临时缓存，突破字节模型中计算与分块大小的绑定瓶颈。
Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models
- 在分块内插入瞬时缓存，实时更新上下文以减少预测滞后
- 16字节分块下性能接近基准模型，推理计算减少3-4倍
- 适合需要低内存高效率的部署场景，如边缘设备或代码生成
无分词器语言模型直接处理字节，分块变体通过聚合连续字节提高效率。但分块大小与计算量、缓存占用和建模质量存在紧密权衡：分块越大，计算越少、缓存越小，但建模质量下降。我们发现这一权衡源于分块延迟：在分块未完全观测前，内部字节预测依赖前一分块的过时表示以维持因果性，分块越大延迟越长。本文提出「临时缓存分块」（Scratchpad Patching, SP），在每一分块内插入临时缓存区，根据下一个字节预测熵触发，动态聚合已见字节信息并刷新上下文。该机制仅在信息密集区域分配计算，支持运行时灵活调整。在自然语言和代码数据集上，SP在相同分块大小下显著提升模型质量；例如，16字节分块时，其性能可匹配或接近原生字节模型，同时将分块缓存缩小16倍，推理计算减少3-4倍。
原文摘要 · Abstract (English)
Tokenizer-free language models eliminate the tokenizer step of the language modeling pipeline by operating directly on bytes; patch-based variants further aggregate contiguous byte spans into patches for efficiency. However, the average patch size chosen at the model design stage governs a tight trade-off: larger patches reduce compute and KV-cache footprint, but degrade modeling quality. We trace this trade-off to patch lag: until a patch is fully observed, byte predictions within it must rely on a stale representation from the previous patch to preserve causality; this lag widens as patches grow larger. We introduce Scratchpad Patching (SP), which inserts transient scratchpads inside each patch to aggregate the bytes seen so far and refresh patch-level context for subsequent predictions. SP triggers scratchpads using next-byte prediction entropy, selectively allocating compute to information-dense regions and enabling post-hoc adjustment of inference-time compute. Across experiments on natural language and code, SP improves model quality at the same patch size; for example, even at $16$ bytes per patch, SP-augmented models match or closely approach the byte-level baseline on downstream evaluations while using a $16\times$ smaller KV cache over patches and $3$-$4\times$ less inference compute.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。