研究不同记忆呈现方式对大模型问答的影响，发现展示形式显著影响得分。
RENDER: Controlling Reader-Facing Evidence in LLM Memory Evaluation
- 设计了五级信息包阶梯，控制输入内容但改变呈现形式。
- 相同内容以聊天风格展示时比原始对话高42.4-72.6分。
- 适合评估记忆系统或做RAG优化的研究者参考。
现有记忆与RAG评估常将模型输入视为实现细节，而实际系统可能以记忆条目、摘要、文本记录或原始对话等形式呈现相同历史。我们提出RENDER基准，固定对话内容但变化读者可见的表达形式。该方法结合五级信息包阶梯，定位答案内容进入输入的时间点，并使用确定性模板模拟ChatGPT风格条目、LangChain摘要、MemGPT式文本记录及原始对话。在500个LongMemEval问题和9个模型上，匹配预算的解析包比仅截取近期对话的原始对话高42.4-72.6分。在部署风格模板下，各模型最佳与最差得分相差24.6-48.8分；主评分器下，7/9模型中ChatGPT风格条目得分高于原始对话。裁判重评保持整体正向效应，但模型层面显著性不一。三个在正式账本包上得分为0%的模型，在自然语言条目上回答正确率可达45.4%-53.4%。该效应在检索噪声下仍存在，并可迁移至HotpotQA，表明记忆/RAG评估应报告或控制读者可见的表达形式。
原文摘要 · Abstract (English)
Memory and RAG evaluations often treat the answering model's input as an implementation detail, even though systems may render the same history as a memory entry, summary, typed record, or raw excerpt. We introduce RENDER, a benchmark control that fixes the conversation while varying the reader-facing artifact. RENDER combines a five-level packet ladder, localizing when answer-bearing content enters the input, with deterministic templates approximating ChatGPT-style entries, LangChain summaries, MemGPT-style typed records, and raw conversation. On 500 LongMemEval questions and nine models, matched-budget resolved packets beat recency-truncated raw dialogue by 42.4-72.6 points. In deployed-style templates, best-worst spread is 24.6-48.8 points per model; under the primary scorer, ChatGPT-style entries have higher point estimates than raw conversation on 7 of 9 models. Judge rescoring preserves the positive aggregate effect, but model-specific significance is mixed. Three models scoring 0 percent on formal ledger packets answer the same facts from natural-language entries at 45.4-53.4 percent. The effect persists under retrieval noise and transfers to HotpotQA, suggesting that memory/RAG evaluations should report or control the reader-facing artifact.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。