用数学推理的对话结构提升模型表现，尤其对小模型和泛化任务有效。
DIMSUM: Discourse in Mathematical Reasoning as a Supervision Module
- 引入数学问题的对话结构作为新监督信号
- 使Llama2-13b性能最高提升160%
- 显著改善模型在分布外数据上的表现，适合小模型研究者
我们研究GSM8k数据集上短文本数学题的推理能力。与Mirzadeh等人（2024）发现一致，当前大模型在该数据集上的进展可能并非源于推理能力提升，而是更广预训练数据分布的暴露。为此，我们提出一种新信息源——对话结构，以帮助数据较少或训练较差的模型更好推理。实验表明，引入对话结构可使Llama2-13b性能最高提升160%。即使对已基本记忆数据集的大模型，加入对话结构仍能提升预测准确率，并显著增强其在分布外样本上的表现。
原文摘要 · Abstract (English)
We look at reasoning on GSM8k, a dataset of short texts presenting primary school, math problems. We find, with Mirzadeh et al. (2024), that current LLM progress on the data set may not be explained by better reasoning but by exposure to a broader pretraining data distribution. We then introduce a novel information source for helping models with less data or inferior training reason better: discourse structure. We show that discourse structure improves performance for models like Llama2 13b by up to 160%. Even for models that have most likely memorized the data set, adding discourse structural information to the model still improves predictions and dramatically improves large model performance on out of distribution examples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。