模型抗噪能力差异源于分词机制，而非架构。
When Is Noise Response Universal? Tokenization as the Hidden Variable in Language Models
- 分词方式决定模型对噪声的敏感度：字符级噪声影响远大于词级。
- 不同架构模型在词级噪声下表现趋同，字符级则分化明显。
- 可通过噪声增强训练提升特定噪声尺度下的鲁棒性，无需额外测试。
文本神经模型在输入含错别字、光学识别错误或漏词等噪声时性能常下降。本文研究了各类模型（句向量模型与仅解码器型大模型）在噪声下的性能衰减率，发现其一致性取决于噪声规模：词级噪声下，不同架构模型衰减曲线高度一致；而字符级噪声下，模型表现显著分化。进一步分析表明，决定因素是训练目标而非架构——8种覆盖6种预训练范式的编码器初始分散，经短时对比学习后均收敛至同一曲线。根源在于分词机制：单个字符修改会迫使分词器重切分周围词汇，扰乱整个词元序列，远比丢失整词影响更剧烈。该发现揭示了抗噪能力的内在机制，可无须实际噪声评估即预测模型鲁棒性，并通过针对性噪声增强训练实现指定噪声尺度下的鲁棒性提升。
原文摘要 · Abstract (English)
The performance of textual neural models often degrades when their inputs are corrupted by noise such as typos, OCR errors, or dropped words. We study the degradation rate across neural models, both sentence embeddings and decoder-only LLMs, and find that how consistent it is depends on the scale of the noise: under word-level noise, models with very different architectures decline along nearly the same curve, while under character-level noise they separate. We further identify the determining factor to be the training objective, not the architecture: eight encoders spanning six pretraining paradigms are scattered initially, and collapse onto a common curve after a short contrastive training recipe. We trace the word/character split to tokenization: a single character edit forces the tokenizer to re-segment the surrounding word, disturbing the token sequence far more than dropping a whole word does. This finding and its underlying mechanism provide a practical means to predict a model's robustness to noise without any noisy evaluation, and to install robustness at a chosen noise scale through noise-augmented training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。