arXiv:2510.18250cs.AI2025-10被引 2

提出ssToken方法,提升大模型微调时的文本质量。

ssToken: Self-modulated and Semantic-aware Token Selection for LLM Fine-tuning

  • 用自身历史模型生成自调节信号,动态选择关键词元。
  • 引入注意力机制捕捉语义重要性,不依赖损失值筛选。
  • 无需额外参考模型,适配多种大模型且训练高效。

数据质量对大语言模型(LLM)监督微调(SFT)至关重要,词元级数据选择因粒度精细而成为研究热点。现有方法存在两大缺陷:一需训练或访问额外参考模型;二仅依赖损失信息筛选,难以保留损失低但语义重要的词元。为此,我们提出ssToken——一种自调节且语义感知的词元选择方法。该方法利用可得的历史模型计算当前模型与之在每个词元上的损失差异,作为自调节信号,使模型能沿优化轨迹自适应选择词元,避免依赖离线训练的参考模型。此外,我们设计了一种基于注意力的语义感知词元重要性评估指标,与损失导向选择正交,提供互补的语义信息以实现更有效的过滤。在多种模型架构与规模下的实验表明,仅使用自调节或语义感知选择即优于全数据微调,二者结合后性能进一步超越已有词元级选择方法,在保持训练效率的同时实现显著性能提升。

原文摘要 · Abstract (English)

Data quality plays a critical role in enhancing supervised fine-tuning (SFT) for large language models (LLMs), and token-level data selection has emerged as a promising direction for its fine-grained nature. Despite their strong empirical performance, existing token-level selection methods share two key limitations: (1) requiring training or accessing an additional reference model, and (2) relying solely on loss information for token selection, which cannot well preserve semantically important tokens that are not favored by loss-based metrics. To address these challenges, we propose ssToken, a Self-modulated and Semantic-aware Token Selection approach. ssToken leverages readily accessible history models to compute the per-token loss difference with the current model, which serves as a self-modulated signal that enables the model to adaptively select tokens along its optimization trajectory, rather than relying on excess loss from an offline-trained reference model as in prior works. We further introduce a semantic-aware, attention-based token importance estimation metric, orthogonal to loss-based selection and providing complementary semantic information for more effective filtering. Extensive experiments across different model families and scales demonstrate that both self-modulated selection and semantic-aware selection alone outperform full-data fine-tuning, while their integration--ssToken--achieves synergistic gains and further surpasses prior token-level selection methods, delivering performance improvements while maintaining training efficiency.

大模型微调词元选择自调节语义感知

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。