用模型内部信息指导大模型训练数据优化，提升效率与效果
Guiding LLM Post-training Data Engineering with Model Internals from Sparse Autoencoders
- 通过稀疏自编码器提取模型内部信号，量化数据多样性、难度和质量
- 在Qwen2.5-Math-1.5B上提升准确率3.00%，节省20%训练步数
- 方法可跨模型规模与算法复用，适合强化学习训练的数据工程
模型内部蕴含丰富的训练数据处理信息，但后训练数据工程仍主要依赖外部信号，忽视了模型内部的内在线索。本文提出SAERL框架，用于大语言模型强化学习（RL）的数据工程。该框架利用稀疏自编码器（SAE）提取的模型内部信息，建模数据的三个内在属性：多样性、难度和质量。分别对应具体操作：在SAE空间中进行适度批混合聚类以控制批次多样性，使用难度代理实现从易到难的课程排序，以及通过质量探测器筛选数据。SAERL在Qwen2.5-Math-1.5B上相比基线GRPO平均准确率提升3.00%，且在达到目标准确率时减少20%训练步数，且在不同模型规模与强化学习算法下均表现一致提升。实验表明SAE能有效跨模型家族与规模迁移，作为轻量级、可复用的数据工程工具。结果证明，模型内部是后训练数据工程的强大且实用的信号源。
原文摘要 · Abstract (English)
Model internals encode rich information about how a large language model (LLM) processes its training data; however, post-training data engineering largely relies on external signals and ignores rich intrinsic signals lying in model internals. We propose SAERL, a data engineering framework for LLM reinforcement learning (RL). It models three intrinsic data properties: diversity, difficulty, and quality, using model internals extracted with Sparse Autoencoder (SAE), an advanced mechanistic interpretability tool. Each property grounds a concrete data engineering operation: SAE-space clustering with moderate batch mixing for batch diversity control, a difficulty proxy for easy-to-hard curriculum ordering, and a quality probe for data filtering. SAERL improves average accuracy by 3.00% over vanilla GRPO and reaches target accuracy with 20% fewer training steps on Qwen2.5-Math-1.5B, with consistent gains across model scales and RL algorithms. Experiments show that SAE transfers effectively across model families and scales, serving as a lightweight and reusable data engineering tool. These results demonstrate that model internals are a powerful and practical source of signals for post-training data engineering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。