实验证明，对话模型的上下文长度影响显著，需重视这一常被忽略的设置。
An Empirical Study on Context Length for Open-Domain Dialog Generation
- 通过控制上下文长度，研究其对模型训练的影响机制。
- 长上下文不总更好，最佳长度因任务而异，存在最优区间。
- 不同对话样本对上下文长度偏好不同，适合个性化处理。
近年来，基于Transformer的开放域对话模型日益流行。这些模型通常将对话历史拼接为上下文表示，但缺乏决定应保留多少轮对话的准则。本文从粗到细提出三个问题：(i) 更长的上下文是否有助于模型训练？(ii) 面对不同长度的对话时，是否需要调整训练中的上下文长度？(iii) 不同对话样本是否对上下文长度有相同偏好？实验结果表明，上下文长度这一常被忽视的设置，在实现Transformer-based对话模型时值得重点关注。
原文摘要 · Abstract (English)
Transformer-based open-domain dialog models have become increasingly popular in recent years. These models typically represent context as a concatenation of a dialog history. However, there is no criterion to decide how many utterances should be kept adequate in a context. We try to figure out how the choice of context length affects the model. We experiment on three questions from coarse to fine: (i) Does longer context help model training? (ii) Is it necessary to change the training context length when dealing with dialogs of different context lengths? (iii) Do different dialog samples have the same preference for context length? Our experimental results show that context length, an often overlooked setting, deserves attention when implementing Transformer-based dialog models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。