用信息论方法自动构建高效、可复用的上下文，提升大模型推理质量。
Directed Information $γ$-covering: An Information-Theoretic Framework for Context Engineering
- 基于定向信息定义上下文覆盖标准，量化因果预测能力。
- 贪心算法保证信息保留率，压缩后仍保持高精度，优于BM25。
- 离线计算一次即可支持所有查询，适合长上下文场景应用。
我们提出一种名为定向信息γ-覆盖（Directed Information γ-covering）的通用冗余感知上下文工程框架。定向信息（DI）是互信息的因果版本，用于衡量块间不对称的预测能力。若满足 $ ext{DI}_{i o j} \ geq H(C_j) - \gamma$，则 $C_i$ 可在γ比特误差内表示 $C_j$。基于此准则，我们将上下文选择建模为γ-覆盖问题，并设计了一个具有理论保障的贪心算法：能保持查询信息在有界松弛内，继承子模集合覆盖的 $(1+\ln n)$ 与 $(1-1/e)$ 近似比，并引入多样性边界。关键优势在于：γ-覆盖构建过程与查询无关，仅需一次离线计算，可跨所有查询分摊开销。在 HotpotQA 上的实验表明，γ-覆盖持续优于 BM25 基线，在上下文压缩和单槽提示选择等困难任务中表现更优。这些结果确立了 DI γ-覆盖作为现代大模型流水线中一个原理清晰、自组织的骨干结构。
原文摘要 · Abstract (English)
We introduce \textbf{Directed Information $γ$-covering}, a simple but general framework for redundancy-aware context engineering. Directed information (DI), a causal analogue of mutual information, measures asymmetric predictiveness between chunks. If $\operatorname{DI}_{i \to j} \ge H(C_j) - γ$, then $C_i$ suffices to represent $C_j$ up to $γ$ bits. Building on this criterion, we formulate context selection as a $γ$-cover problem and propose a greedy algorithm with provable guarantees: it preserves query information within bounded slack, inherits $(1+\ln n)$ and $(1-1/e)$ approximations from submodular set cover, and enforces a diversity margin. Importantly, building the $γ$-cover is \emph{query-agnostic}: it incurs no online cost and can be computed once offline and amortized across all queries. Experiments on HotpotQA show that $γ$-covering consistently improves over BM25, a competitive baseline, and provides clear advantages in hard-decision regimes such as context compression and single-slot prompt selection. These results establish DI $γ$-covering as a principled, self-organizing backbone for modern LLM pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。