将推理轨迹前置可显著提升长文本推理能力
Trace as State: Reasoning Traces as Conditional States for Long-Context Transformers
- 用推理轨迹作为任务状态，提前放置于上下文前引导重读
- 在三个数据集上26/27组合中表现优于后置轨迹的对照组
- 对DeepSeek和GLM模型提升明显，最高准确率达100%
Transformer 按因果顺序处理信息，但长文本推理可能依赖后期才确定的任务状态。我们通过条件状态更新任务形式化这一矛盾。对于因果状态更新机制，先提供条件可使最坏情况下内存需求指数级降低。受此启发，我们提出 Trace as State：将收集的推理轨迹作为任务状态的文本代理，提前置于长上下文块之前，在重新阅读时引导信息利用。我们在三个模型和三个长文本数据集上进行实验，结果表明，在26/27个组合中，Trace as State优于作为对照的 Trace Append（相同状态代理但置于上下文之后）。在 GraphWalks Parents 数据集上，DeepSeek V4 Pro Preview 的精确匹配率从初始的29.2%、追踪后置的43.0%提升至81.8%；GLM-5.2 从66.4%、83.2%提升至100.0%。结果证明，将轨迹前置可在保持因果结构的同时显著增强长文本推理能力。
原文摘要 · Abstract (English)
Transformers process information causally, but long-context reasoning may depend on task state discovered only later. We formalize this mismatch through conditional state update tasks. For causal state update processors, providing the condition first can require exponentially less memory in the worst case than providing it last. Motivated by this principle, we introduce Trace as State. We use collected reasoning traces as a textual proxy for task state and place it before the long-context block on a fresh pass, allowing information derived previously to guide rereading. We conduct extensive experiments on Trace as State and Trace Append, a matched control that uses the same task state proxy but put it after the context. Across three models and three long-context datasets, Trace as State outperforms Trace Append in 26 of 27 reported combinations of model, task, and metric. On GraphWalks Parents, exact match lifts DeepSeek V4 Pro Preview from 29.2% on the initial pass and 43.0% with Trace Appendto 81.8% with Trace as State, and from 66.4% and 83.2% to 100.0% for GLM-5.2. These results show that placing traces before the context can improve long-context reasoning while retaining the causal transformer structure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。