用时间约束训练语言模型，避免金融预测中因提前看到结果导致的偏差。
DatedGPT: Preventing Lookahead Bias in Large Language Models with Time-Aware Pretraining
- 在每年截止数据下从头训练12个1.3B模型，防止知识泄露。
- 在6.1万条公司日新闻上实现年化夏普比率3.20，无前瞻偏差。
- 提供可交互网页，对比不同年份模型的预测表现，适合金融与模型评估研究者。
大规模语言模型在互联网级数据上预训练时，可能因训练数据中已包含未来结果而产生前瞻偏差。为解决此问题，我们提出DatedGPT，一个由12个1.3B参数模型组成的系列，每个模型均从零开始训练，使用约1000亿个标记的数据，且严格按年份截断（2013–2024）。同时构建了基于各年度文档的指令数据集DatedInstruct，以防止微调阶段泄露信息。模型在同类开源模型中表现相当，困惑度探测证实各模型知识受限于其截断年份。在61,000条公司日新闻标题上的股票收益预测任务中，DatedGPT-instruct 在无前瞻偏差设置下达到年化夏普比率3.20。相比之下，存在前瞻偏差的模型每标准差额外获得26.4基点的超额收益，统计显著性达1%。该系列模型使前瞻偏差的直接分析成为可能。我们提供了交互式网页演示，用户可查询并比较不同截断年份模型的输出，网址为www.datedgpt.com。
原文摘要 · Abstract (English)
Large language models pretrained on internet-scale data risk lookahead bias in forecasting tasks, as they may have already seen the true outcome during training. To address this, we present DatedGPT, a family of twelve 1.3B-parameter language models trained from scratch on approximately 100 billion tokens each with strict annual data cutoffs spanning 2013 to 2024, together with DatedInstruct, an instruction dataset grounded in each year's documents to prevent leakage during post-training. The models are competitive with open models of similar scale, and perplexity-based probing confirms that each model's knowledge is bounded by its cutoff year. On stock return prediction over 61,000 firm-day news headlines, DatedGPT-instruct achieves an annualised Sharpe ratio of $3.20$ under the lookahead-bias-free setup. Lookahead-biased models, whose training data covers the outcome period, add a lookahead premium of $26.4$ b.p. per standard deviation, significant at the 1% level. The series thus enables direct analysis of lookahead bias in financial forecasting. We provide an interactive web demo that allows users to query and compare responses from models across different cutoff years, available at www.datedgpt.com.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。