padding tokens会悄悄影响大模型表现，不容忽视。
Silent Tokens, Loud Effects: Padding in LLMs
- 通过控制填充量，系统测试了不同模型的响应变化
- 少量填充就导致表征偏移、生成质量下降和安全机制削弱
- 提醒部署时必须严格处理填充，否则可能引发严重问题
在大型语言模型（LLMs）的批处理推理中，填充令牌（padding tokens）被广泛用于统一序列长度。尽管应完全屏蔽，但实现错误可能导致其影响计算，而这种影响的程度尚未明确。本文系统研究了三种开源模型族（Llama、Gemma、Qwen）在插入可控数量填充后的表现，从四个维度评估：激活值、生成质量、偏差和安全性。结果表明，即使少量填充也会引起隐藏表示偏移，降低小型模型的生成质量，以不可预测方式改变偏差，并削弱安全防护机制。这些发现表明，填充并非无害细节，而是需在部署中谨慎处理的鲁棒性风险。
原文摘要 · Abstract (English)
Padding tokens are widely used in large language models (LLMs) to equalize sequence lengths during batched inference. While they should be fully masked, implementation errors can cause them to influence computation, and the extent of this influence is not well understood. We systematically study this effect across three open-source model families (Llama, Gemma, Qwen), inserting controlled amounts of padding and evaluating outcomes along four axes: activations, generation quality, bias, and safety. Even small amounts of padding shift hidden representations, degrade quality in smaller models, alter bias in unpredictable ways, and weaken safety guardrails. These findings demonstrate that padding is not a harmless detail but a robustness risk that must be carefully handled in deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。