指出语言模型效率评估中的选择偏差问题
Note on Selection Bias in Observational Estimates of Algorithmic Progress
- 发现算法质量评估受计算资源选择影响
- 高效率模型更可能被选中投入更多算力
- 适合关注算法评估方法的学者阅读
Ho 等人（2024）尝试通过观测语言模型随时间变化的损失和计算量，估算算法进步程度。他们认为，随着时间推移，语言模型在固定算力下的损失持续下降，表明算法效率提升。本文指出该估计策略存在潜在方法论问题：若算法质量部分为不可观测的隐变量，且算力配置内生于算法质量，则所估算的算法质量会受到选择偏差污染。
原文摘要 · Abstract (English)
Ho et. al (2024) attempts to estimate the degree of algorithmic progress from language models. They collect observational data on language models' loss and compute over time, and argue that as time has passed, language models' algorithmic efficiency has been rising. That is, the loss achieved for fixed compute has been dropping over time. In this note, I raise one potential methodological problem with the estimation strategy. Intuitively, if part of algorithmic quality is latent, and compute choices are endogenous to algorithmic quality, then resulting estimates of algorithmic quality will be contaminated by selection bias.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。