用稀疏自编码器增强目标语言特征，无需训练即可提升多语言模型表现。
Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inference
- 通过预训练稀疏自编码器识别并强化目标语言相关特征。
- 在XCOPA、XNLI等数据集上平均准确率提升超5个百分点。
- 适用于需要快速适配新语言的场景，尤其适合资源有限时使用。
多语言大模型在不同语言上的表现差异显著，而现有适配方法通常需要参数更新和大量多语言训练数据。本文提出一种推理阶段的多语言引导方法，利用预训练的稀疏自编码器（SAE）识别并强化与目标语言相关的特征。基于多语言平行句，比较不同语言下的SAE激活值，筛选出每个目标语言对应的少量层特定特征。这些特征被解码为引导信号，并注入模型隐藏状态，无需额外训练。在Gemma-3-12B-it上实验显示，该方法在XCOPA上平均准确率提升10.9个百分点，在XNLI上提升5.3个百分点，在MGSM上提升1.9个百分点。
原文摘要 · Abstract (English)
Multilingual large language models exhibit substantial performance differences across languages, while existing adaptation methods often require parameter updates and considerable multilingual training data. We propose an inference-time multilingual steering method that uses pretrained sparse autoencoders to identify and strengthen target-language-related features. Using multilingual parallel sentences, we compare SAE activations across languages and select a small number of layer-specific features associated with each target language. These features are decoded into steering signals and injected into the model's hidden states without additional training. Experiments with Gemma-3-12B-it show average accuracy improvements of 10.9 percentage points on XCOPA, 5.3 points on XNLI, and 1.9 points on MGSM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。