通过训练数据溯源，揭示大模型社会推理与理工推理依赖不同语料区域。
Capability Provenance in Language Models: A Case Study in Social Reasoning
- 用梯度归因法追踪语料中每篇文档对模型表现的影响。
- 社会推理和理工推理分别依赖截然不同的语料区域，推理能力差异更明显。
- 可针对性遗忘高贡献主题，验证因果关系，适合模型可解释性研究者。
我们利用训练数据溯源作为可解释工具，识别OLMo3-7B模型中支持社会推理与理工推理的预训练语料区域。训练数据溯源衡量每篇训练文档对模型在基准测试上预测结果的影响程度，但文档级得分噪声较大，难以精确定位能力来源。因此，我们在去重后的Dolma3语料集基础上构建工作集，采用基于梯度的归因方法（TrackStar via Bergson），并按WebOrganizer的24种格式×24个主题分类（共576个类别）聚合影响度，通过2×2实验设计对比：社会领域（SocialIQA）vs. 理工领域（ARC-Challenge）与推理能力（reasoning）vs. 知识能力（knowledge）；结果显示，社会推理与理工推理依赖的语料区域具有本质差异，且这种区分在推理层面比知识层面更显著。通过目标机器遗忘实验进行部分因果验证：删除高归因主题（如文学类对SocialIQA）导致对应基准性能下降，优于同主题随机基线。该结论在另一开源模型Comma v0.1 7B-2T（Common Pile）与DCLM-Baseline-7B（DataComp-LM）上也成立，但溯源图谱具生态系统特异性。所有代码、数据、影响分数及检查点已开源至https://github.com/eilab-gt/capabilibara 和 https://huggingface.co/HCAI-Lab。
原文摘要 · Abstract (English)
We use training-data attribution as an interpretable tool for capability discovery, mapping which regions of the pretraining corpus support social-reasoning versus STEM-reasoning in OLMo3-7B. Training-data attribution measures how strongly each training document influences a model's predictions on a benchmark, but document-level scores are too noisy to identify which corpus regions support which capabilities. We compute gradient-based attribution (TrackStar via Bergson) over a working set drawn from the de-duplicated Dolma3 mix, aggregate influence across WebOrganizer's 24-format x 24-topic taxonomy (576 bins), and contrast benchmark pairs in a 2x2 design that varies domain (social vs. STEM) and capability type (reasoning vs. knowledge): SocialIQA and MMLU Social Sciences against ARC-Challenge and MMLU STEM. Social and STEM reasoning draw on qualitatively distinct corpus regions, and the contrast is sharper at the reasoning level than at the knowledge level. Targeted machine unlearning provides partial causal validation: forgetting high-attribution topics (e.g., Literature for SocialIQA) degrades the aligned benchmark more than within-topic random baselines. We validate on other open-data model, Comma v0.1 7B-2T (Common Pile) and DCLM-Baseline-7B (DataComp-LM): causal selectivity holds on both models, while the provenance map is ecosystem-specific. We open-source all code, data artifacts, influence scores, and checkpoints at https://github.com/eilab-gt/capabilibara and https://huggingface.co/HCAI-Lab.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。