用信息论方法筛选有用上下文，让大模型更精准理解问题。
Entropic Context Shaping: Information-Theoretic Filtering for Context-Aware LLM Agents
- 通过答案分布变化衡量上下文实际帮助程度，而非简单比对词语。
- 在多轮对话中选择关键片段，准确率比传统方法提升71.83%。
- 适合需要精细上下文管理的智能助手、客服机器人等场景。
大型语言模型代理的上下文工程需区分实用信息与干扰项。我们提出熵性上下文塑造（ECS），一种基于信息论的框架，通过模型答案分布向正确答案的偏移来衡量上下文效用。不同于依赖词重合的词汇相似度方法，ECS捕捉的是语用实用性——某段文本是否真正有助于回答问题。我们将效用形式化为答案概率的有符号变化，并提供理论分析表明无关任务更新导致近零分布偏移。在多轮上下文选择任务上，使用LongMemEval（会话级）和LoCoMo（轮次级）基准进行评估。在细粒度轮次选择中，采用Llama-3.1-8B的ECS达到F1=0.265，相较TF-IDF（F1=0.154）实现71.83%的相对提升，证明当精确选择上下文至关重要时，语用效用优于词汇相似性。代码与数据见附录。
原文摘要 · Abstract (English)
Context engineering for large language model (LLM) agents requires distinguishing pragmatically useful information from misleading distractors. We introduce Entropic Context Shaping (ECS), an information-theoretic framework that measures context utility via the shift in the model's answer distribution toward the correct answer. Unlike lexical similarity methods that rely on word overlap, ECS captures pragmatic utility -- whether a passage actually helps answer the question. We formalize utility as the signed change in answer probability and provide theoretical analysis showing that task-irrelevant updates yield near-zero distribution shift. We evaluate on multi-turn context selection tasks using LongMemEval (session-level) and LoCoMo (turn-level) benchmarks. On fine-grained turn selection, ECS with Llama-3.1-8B achieves F1=0.265, a 71.83% relative improvement over TF-IDF (F1=0.154), demonstrating that pragmatic utility outperforms lexical similarity when precise context selection matters. Code and data are available in the supplementary materials.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。