研究非正式文本如何降低NLI模型准确率并提出针对性修复方法。
When Informal Text Breaks NLI: Tokenization Failure, Distribution Shift, and Targeted Mitigations
- 分析四种非正式化变换对模型的影响，发现表情符号破坏分词信号，噪声词缺乏训练数据。
- 通过预处理和数据增强结合，使ELECTRA在混合变换下准确率达88.93%，且不损害原数据表现。
- 适合关注模型鲁棒性、社交媒体文本理解或对抗性测试的研究者参考。
我们研究了在SNLI和MultiNLI上对ELECTRA-small（14M）和RoBERTa-large（355M）施加四种变换（俚语替换、表情符号替换、Gen-Z填充词、组合）时，非正式表达如何降低NLI准确率。俚语替换仅导致最多1.1个百分点的下降：因词汇在WordPiece覆盖范围内，分词未丢失信号。表情符号将内容词替换为Unicode字符，ELECTRA的WordPiece分词器将其映射为[UNK]，造成输入信号破坏（93.6%的表情符号样本含至少一个[UNK]，平均每例2.91个）。噪声词（如no cap, deadass, tbh）虽在词汇表中，但未出现在训练数据中，模型却赋予其错误推理权重。两种失效模式需不同干预：预处理可恢复表情符号准确性；数据增强通过训练中引入噪声样本缓解问题。两者结合使ELECTRA在SNLI的混合变换下达到88.93%准确率（原始75.88%），且在干净文本上无显著下降。与GPT-4o-mini零样本相比，未修复的ELECTRA在变换后显著更差（p < 0.0001）；经修复的ELECTRA在所有SNLI变体上超越它，并在MultiNLI上达到统计等效。
原文摘要 · Abstract (English)
We study how informal surface forms degrade NLI accuracy in ELECTRA-small (14M) and RoBERTa-large (355M) across four transforms applied to SNLI and MultiNLI: slang substitution, emoji replacement, Gen-Z filler tokens, and their combination. Slang substitution (replacing formal words with informal equivalents, e.g., "going to" -> "gonna", "friend" -> "homie") causes minimal degradation (at most 1.1pp): slang vocabulary falls largely within WordPiece coverage, so the tokenizer handles it without signal loss. Emoji replaces content words with Unicode characters that ELECTRA's WordPiece tokenizer maps to [UNK], destroying the input signal before any learned parameters see it (93.6% of emoji examples contain at least one [UNK], mean 2.91 per example). Noise tokens (no cap, deadass, tbh) are fully in-vocabulary but absent from NLI training data, consistent with the model assigning them inferential weight they do not carry. The two failure modes respond to different interventions: preprocessing recovers emoji accuracy by normalizing text before tokenization; augmentation handles noise by exposing the model to noise-bearing examples during training. A hybrid of both achieves 88.93% on the combined variant for ELECTRA on SNLI (up from 75.88%), with no statistically significant drop on clean text. Against GPT-4o-mini zero-shot, unmitigated ELECTRA is significantly worse on transformed variants (p < 0.0001); hybrid ELECTRA surpasses it across all SNLI variants and reaches statistical parity on MultiNLI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。