用概率语言树统一压缩、决策与推理复用,提升效率。
Probabilistic Language Tries: A Unified Framework for Compression, Decision Policies, and Execution Reuse
- 构建概率语言树,显式表示生成模型的前缀结构。
- 推理成本从O(n²)降至期望p_r×O(log N) + (1−p_r)×O(n²)。
- 适用于棋类、搜索、机器人控制等场景,适合高效推理系统开发者。
我们提出概率语言树(PLT),一种统一表示框架,显式表达任意序列生成模型隐含的前缀结构。通过为每条出边分配对应标记或动作的条件概率,PLT同时实现:(i) 基于频率加权区间编码的最优无损压缩,泛化算术编码至模型依赖分布;(ii) 序列决策问题(如棋类、搜索、机器人控制)的策略表示;(iii) 通过结构化检索实现重复推理查询的缓存复用,而非完整模型执行。核心技术成果为先验引导缓存定理:在平稳生成分布下,当查询次数低于随先验集中度增长的阈值时,PLT引导缓存的期望推理成本严格低于任何基于经验频率的缓存。该方法将Transformer的O(n²)注意力成本转化为期望成本p_r × O(log N) + (1 − p_r) × O(n²),其中p_r为先验估计的复用概率,N为存储规模。我们进一步提出混合压缩架构,将数据集分解为被PLT覆盖的主体部分与稀疏残差存储,连接算术编码、柯尔莫戈洛夫型程序表示与率失真理论。框架在国际象棋、网络搜索、机器人控制、组织工作流及大模型推理中得到验证,证明压缩、决策与计算复用均源于序列空间上的单一概率测度。
原文摘要 · Abstract (English)
We introduce probabilistic language tries (PLTs), a unified representation that makes explicit the prefix structure implicitly defined by any generative model over sequences. By assigning to each outgoing edge the conditional probability of the corresponding token or action, a PLT simultaneously serves as: (i) an optimal lossless compressor via frequency-weighted interval encoding, generalizing arithmetic coding to model-conditioned distributions; (ii) a policy representation for sequential decision problems including games, search, and robotic control; and (iii) a memoization index that lets repeated inference queries be answered by structured retrieval rather than full model execution. The central technical result is a prior-guided caching theorem: under a stationary generative distribution, a PLT-guided cache achieves strictly lower expected inference cost than any empirical-frequency cache for all query counts below a threshold that grows with the concentration of the prior. This converts O(n^2) transformer attention cost into an expected cost of p_r * O(log N) + (1 - p_r) * O(n^2), where p_r is the prior-estimated reuse probability and N is the artifact store size. We further introduce a hybrid compression architecture decomposing any dataset into a PLT-covered majority and a sparse residual store, connecting arithmetic coding with Kolmogorov-style program representations and rate-distortion theory. We instantiate the framework across chess, web search, robotics, organizational workflows, and LLM inference, demonstrating that compression, decision making, and computational reuse are all derived from a single probability measure on sequence space.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。