用棋局对训练小模型，发现高分靠记忆而非理解
Generalization or Memorization? Brittleness Testing for Chess-Trained Language Models
- 用(局面,最佳着法)对训练2500万参数小模型
- 在600题将死难题上超越30亿参数模型
- 验证框架让普通大模型准确率提升至21.2%
近期研究将语言模型微调用于棋类数据，并以高分成绩声称模型掌握了棋规、能专业对弈或生成专家级解释。我们训练了仅基于(局面,最佳着法)对的2500万参数字符级模型KinGPT，其在600题将死难题集上的表现超过30亿参数的ChessGPT，且在20主题谜题基准上优于40亿参数的C1-4B。我们检验了现有文献中关于棋类语言模型的多项主张，指出其优异表现主要源于模式匹配。此外，我们展示了一个验证器内循环框架LLM-Modulo，可将RedPajama 3B在将死难题上的最佳着法准确率从1.2%提升至21.2%，着法生成有效性从19.3%提升至95.3%，效果接近ChessGPT在棋类语料上的微调，但成本仅为后者的极小部分。结果表明，将通用大模型与外部验证器结合，是特定领域任务更灵活的替代方案。所有训练/评估代码、数据集、谜题样例及KinGPT模型检查点均已开源，确保可复现性。
原文摘要 · Abstract (English)
Recent work has fine-tuned language models on chess data and reported high benchmark scores as evidence that the resulting models can understand the rules of chess, play full chess games at a professional level, or generate human-readable explanations grounded in expert knowledge. We train KinGPT, a 25M-parameter character-level language model trained only on (position, best-move) pairs, who exceeds 3B-parameter ChessGPT on a 600-puzzle mate-in-N suite and 4B-parameter C1-4B over a 20-theme puzzle benchmark. We examine several claims made in existing literature regarding chess-trained language models and assert that their impressive benchmark performance is largely explained by pattern-matching. We also demonstrate how LLM-Modulo, a verifier-in-the-loop framework, raises RedPajama 3B's best move accuracy from 1.2% to 21.2% and move generation validity from 19.3% to 95.3% on mate-in-N chess puzzles, comparable to gains achieved from ChessGPT's fine-tuning on chess-specific web corpora at a fraction of the cost. Our results illustrate how pairing a general LLM with an external verifier offers a more flexible alternative to directly training on synthetic data for well-defined domains. We open source all training/evaluation code, datasets, puzzle samples, and KinGPT model checkpoints for reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。