通过筛选训练数据中的词语，高效抑制语言模型的不良能力。
Shaping capabilities with token-level data filtering
- 用词级数据过滤替代文档级过滤，更精准地削弱有害能力。
- 大规模模型下，词级过滤可使有害任务计算量降低7000倍。
- 即使标签有噪声，只要预训练算力充足，过滤仍有效。
当前减少语言模型不良能力的方法多为事后补救，易被攻击者绕过。一个自然的替代方案是在预训练阶段就塑造模型能力。在移除医学相关能力的代理任务中，我们发现仅对预训练数据进行词级过滤就极为有效、鲁棒且可规模化。受数据归因研究启发，我们证明词级过滤比文档级过滤更优，在同等削弱有害能力的同时，对良性能力的损害更低。在训练跨度达两个数量级的模型后，我们发现过滤效果随规模提升：最大模型在遗忘领域实现7000倍的计算延迟下降。我们还表明，经过词级过滤训练的模型仍可在遗忘领域保持对齐。此外，我们提出一种基于稀疏自编码器标注词的方法，并蒸馏出低成本高精度分类器。最后，我们证明在足够大的预训练算力下，过滤对噪声标签也具有鲁棒性。
原文摘要 · Abstract (English)
Current approaches to reducing undesired capabilities in language models are largely post hoc, and can thus be easily bypassed by adversaries. A natural alternative is to shape capabilities during pretraining itself. On the proxy task of removing medical capabilities, we show that the simple intervention of filtering pretraining data is highly effective, robust, and inexpensive at scale. Inspired by work on data attribution, we show that filtering tokens is more effective than filtering documents, achieving the same hit to undesired capabilities at a lower cost to benign ones. Training models spanning two orders of magnitude, we then demonstrate that filtering gets more effective with scale: for our largest models, token filtering leads to a 7000x compute slowdown on the forget domain. We also show that models trained with token filtering can still be aligned on the forget domain. Along the way, we introduce a methodology for labeling tokens with sparse autoencoders and distilling cheap, high-quality classifiers. We also demonstrate that filtering can be robust to noisy labels with sufficient pretraining compute.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。