将文档表示合并后推理，提升长文本问答效率与效果。
Studying the Soupability of Documents in State Space Models
- 独立编码文档后平均合并表示，实现模块化推理
- 在RACE和QuALITY上优于传统拼接法，多跳问答性能提升显著
- 支持数百文档推理，大幅降低推理开销，适合大规模语料
我们研究结构化状态空间模型（SSMs）的隐藏状态是否可在事后合并以支持下游推理。受模型“炖煮”启发，提出文档“炖煮”策略：文档独立编码后，通过平均等简单操作将其表示聚合为单一上下文状态。该方法支持模块化编码与复用，无需对每个查询重新处理完整输入。实验表明，微调后的Mamba2模型使用炖煮表示，在多跳问答、稀疏检索和长文档推理任务中表现优于或媲美传统整体编码方法。例如，在RACE和QuALITY长文档问答基准上，该方法显著优于传统拼接方式。关键优势在于，这种模块化设计可扩展至数百文档，同时大幅降低推理成本，为大规模语料推理开辟新可能。
原文摘要 · Abstract (English)
We investigate whether hidden states from Structured State Space Models (SSMs) can be merged post hoc to support downstream reasoning. Inspired by model souping, we study document souping, a strategy where documents are encoded independently, and their representations are pooled, via simple operations like averaging, into a single context state. This approach enables modular encoding and reuse without reprocessing the full input for each query. We demonstrate that finetuned Mamba2 models with souped representations achieve competitive or superior performance across multi-hop QA, sparse retrieval, and long-document reasoning tasks compared to the standard monolithic encoding approach. For example, on the RACE and QuALITY benchmarks for long document question answering, this method substantially outperforms a traditional concatenation approach. Crucially, this modular design scales to hundreds of documents while delivering substantial savings in inference cost, unlocking new possibilities for large-scale corpus reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。