提出遗忘曲线方法，更可靠评估长文本模型的记忆能力。
Forgetting Curve: A Reliable Method for Evaluating Memorization Capability for Long-context Models
- 用遗忘曲线替代传统评测，不依赖提示词和数据集。
- 验证了变压器架构扩展的有效性，质疑RNN/SSM模型实际记忆长度。
- 适用于任意模型规模，结果稳定且可复现。
许多近期工作致力于扩展语言模型的有效上下文长度，现有方法、任务与基准用于衡量模型的有效记忆长度。然而，通过深入调查，我们发现当前对模型记忆能力的评估存在局限。本文系统梳理了这些局限，并提出一种新方法——遗忘曲线，用于衡量长上下文模型的记忆能力。结果显示，该方法对测试语料和实验设置具有鲁棒性，无需依赖提示词，且可适用于任意模型规模。我们在涵盖变压器与RNN/SSM架构的多种模型上应用该方法，实证支持了变压器扩展技术的有效性，同时对RNN/SSM模型的实际有效长度提出质疑。我们还对比了该方法与现有基准及常见指标在不同模型上的表现。代码与结果详见：https://github.com/1azybug/ForgettingCurve。
原文摘要 · Abstract (English)
Numerous recent works target to extend effective context length for language models and various methods, tasks and benchmarks exist to measure model's effective memorization length. However, through thorough investigations, we find limitations for currently existing evaluations on model's memorization capability. We provide an extensive survey for limitations in this work and propose a new method called forgetting curve to measure the memorization capability of long-context models. We show that forgetting curve has the advantage of being robust to the tested corpus and the experimental settings, of not relying on prompts and can be applied to any model size. We apply our forgetting curve to a large variety of models involving both transformer and RNN/SSM based architectures. Our measurement provides empirical evidence for the effectiveness of transformer extension techniques while raises questions for the effective length of RNN/SSM based models. We also examine the difference between our measurement and existing benchmarks as well as popular metrics for various models. Our code and results can be found at https://github.com/1azybug/ForgettingCurve.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。