用神经元激活指导数据选择,实现无标注大模型自进化
Neuron-Aware Data Selection for Annotation-Free LLM Self-Distillation

- 通过神经元激活筛选训练数据并构建教师上下文
- 在专业领域任务上提升性能,同时保持跨域泛化能力
- 适合无外部标注或在线反馈的场景,区别于传统强化学习
在缺乏真实交互反馈或人工标注监督的情况下,对大型语言模型进行后训练仍具挑战性,尤其在专家标注成本高昂的专业领域。近期无标注自演化方法利用模型自身输出作为监督信号,通过额外上下文构建教师,并基于多轮推理结果的多数投票生成伪标签。然而这些方法存在缺陷:SFT和GRPO变体易导致域外性能下降,而基于奖励的在线强化学习会放大校准误差。本文提出神经元感知的在线自蒸馏(Neuron-OPSD),一种以数据为中心的无标注自蒸馏框架,利用内部神经元激活指导训练数据选择与教师上下文构造。模型通过从教师分布进行在线蒸馏训练,全程无需真实标签。在多个专业领域基准测试中,Neuron-OPSD在提升域内任务表现的同时,维持了跨域泛化能力,并缓解了校准崩溃问题。该框架特别适用于在线交互或外部监督成本高或不可行的场景,与依赖记录奖励轨迹的离线强化学习方法在概念上相区分。
原文摘要 · Abstract (English)
Post-training large language models (LLMs) without real-world interaction feedback or human-labeled supervision remains challenging, particularly in specialized domains where expert annotations are costly to obtain. Recent annotation-free self-evolution methods address this by using the model's own outputs as supervision signals, constructing a teacher via additional context and aggregating predictions across multiple rollouts through majority voting to produce pseudo-labels. However, these approaches are not without drawbacks: SFT- and GRPO-based variants suffer out-of-domain performance degradation, while reward-based on-policy RL inflates calibration error. In this paper, we propose Neuron On-Policy Self-Distillation (Neuron-OPSD), a data-centric framework for annotation-free self-distillation that leverages internal neuron activations to guide both training-data selection and teacher context construction. The model is then trained via on-policy distillation from the teacher distribution, requiring no ground-truth labels at any stage. Across specialized-domain benchmarks, Neuron-OPSD improves in-domain task performance while preserving cross-domain generalization and mitigating calibration collapse over prior annotation-free baselines. This framework is particularly relevant to settings where online interaction or external supervision is costly or infeasible, and is conceptually distinct from offline RL approaches that rely on logged, reward-labeled trajectories.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。