提出新评估方法，让模型预测更长序列
Provable Length Generalization in Sequence Prediction via Spectral Filtering
- 用谱滤波设计可证明泛化能力的算法
- 在线性动态系统上实现长度外推
- 适合研究模型外推机制的研究者
我们研究序列预测中的长度泛化问题，引入一种新性能度量——非对称遗憾（Asymmetric-Regret），用于衡量学习者在上下文长度受限时相对于具有更长上下文的基准预测器的损失。通过谱滤波视角分析该问题，提出一种基于梯度的学习算法，在线性动态系统上实现了可证明的长度泛化能力。最后的验证实验结果与理论预期一致。
原文摘要 · Abstract (English)
We consider the problem of length generalization in sequence prediction. We define a new metric of performance in this setting -- the Asymmetric-Regret -- which measures regret against a benchmark predictor with longer context length than available to the learner. We continue by studying this concept through the lens of the spectral filtering algorithm. We present a gradient-based learning algorithm that provably achieves length generalization for linear dynamical systems. We conclude with proof-of-concept experiments which are consistent with our theory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。