发现大模型压缩上下文会丢失用户重要指令，提出新方法大幅提高指令保留率。
Lost in Compaction: Evaluating Side-Constraint Loss under Context Compaction
- 设计SC-aware提取器，与压缩模块并行运行，无需修改主模型
- 现有压缩器平均仅保留17%的关键约束指令，性能反而不如不压缩
- 适用于长对话、智能体任务等场景，适合关注上下文完整性研究者
当上下文窗口压力增大时，大模型系统会压缩先前内容以维持任务进行。我们识别出一类用户指令——会话约束（Session Constraints, SCs），如‘在确认前不要删除任何邮件’，这类指令旨在约束整个会话中的行为，但在压缩过程中被无声丢弃。为量化此损失，我们提出COMPINT评估套件，评估压缩器在三种长上下文场景下的表现：多轮对话、智能体轨迹和长周期研究。结果显示，当前压缩器平均仅保留17%的注入SCs，多数表现劣于无压缩情况。保留率受压缩器、提示词、上下文长度、指令表述方式及插入位置显著影响，表明该损失具有系统性。我们提出一个SC-aware提取器，作为即插即用模块与压缩器并行运行，在所有三个场景中实现超过90%的指令保留率，且无需修改压缩器或大模型。COMPINT评估套件及实现代码已开源。
原文摘要 · Abstract (English)
When the context window is under pressure, LLM systems compact prior context to continue ongoing tasks. We identify a class of user-issued instructions, Session Constraints (SCs), such as "do not delete any emails until I confirm," that are meant to constrain LLM's behavior for the remainder of a session but are silently dropped during compaction. To quantify this loss, we introduce COMPINT, an evaluation suite that evaluates compactors across three long-context scenarios: multi-turn chat, agentic trajectory, and long-horizon research. Current compactors retain only 17% of injected SCs on average, and most perform worse than running the same task without compaction. Retention varies sharply with compactor, prompt, context length, SC phrasing, and injection location, showing that the loss is systematic rather than tied to any single setting. We propose an SC-aware extractor that runs alongside the compactor as a plug-and-play module, achieving over 90% retention across all three scenarios without modifying the compactor or LLM. The COMPINT evaluation suite and accompanying implementation are available at https://github.com/ZhiqiEliWang/compaction-integrity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。