分析了变压器模型在随机推理下的鲁棒性，发现小模型最稳定，大模型擅长记忆任务。
Dropout Robustness and Cognitive Profiling of Transformer Models via Stochastic Inference
- 用蒙特卡洛丢弃法对19个模型做100次随机前向传播，评估推理时稳定性。
- 53%模型在标准丢弃下准确率下降超10%，最大降幅达24个百分点。
- 发现记忆类任务对丢弃更敏感，84%模型表现偏重记忆能力。
基于Transformer的语言模型广泛用于推理，但其在推理阶段随机性下的行为仍缺乏系统研究。尽管训练中常用丢弃，但通过蒙特卡洛采样进行的推理阶段丢弃影响尚未在各类架构中系统评估，限制了模型在不确定性感知应用中的可靠性理解。本研究使用MC Dropout对19个Transformer模型进行分析，每样本执行100次随机前向传播。将丢弃鲁棒性定义为在随机推理下保持高准确率与稳定预测的能力，以各次运行准确率的标准差衡量。提出认知分解框架，将性能拆分为记忆与推理成分。实验涵盖五种丢弃配置，共完成95次独立评估，覆盖1,000个样本。结果揭示显著的架构差异：小型模型表现出完美预测稳定性，中等规模模型波动明显，中型模型整体性能最佳，大型模型在记忆任务中占优。关键发现：53%的模型在基线MC Dropout下出现严重准确率下降，任务专用模型最高损失达24个百分点，表明此类架构不适用于不确定性量化。不对称效应显现：高丢弃率使记忆准确率下降27个百分点，而推理仅降1个百分点，说明记忆任务依赖于稳定的表征，易受丢弃破坏。84%的模型呈现记忆偏向性能。这是首个针对Transformer的完整MC Dropout基准，揭示丢弃鲁棒性与模型规模无关，且高度依赖架构。认知画像框架为不确定性感知应用中的模型选择提供可操作指导。
原文摘要 · Abstract (English)
Transformer-based language models are widely deployed for reasoning, yet their behavior under inference-time stochasticity remains underexplored. While dropout is common during training, its inference-time effects via Monte Carlo sampling lack systematic evaluation across architectures, limiting understanding of model reliability in uncertainty-aware applications. This work analyzes dropout-induced variability across 19 transformer models using MC Dropout with 100 stochastic forward passes per sample. Dropout robustness is defined as maintaining high accuracy and stable predictions under stochastic inference, measured by standard deviation of per-run accuracies. A cognitive decomposition framework disentangles performance into memory and reasoning components. Experiments span five dropout configurations yielding 95 unique evaluations on 1,000 samples. Results reveal substantial architectural variation. Smaller models demonstrate perfect prediction stability while medium-sized models exhibit notable volatility. Mid-sized models achieve the best overall performance; larger models excel at memory tasks. Critically, 53% of models suffer severe accuracy degradation under baseline MC Dropout, with task-specialized models losing up to 24 percentage points, indicating unsuitability for uncertainty quantification in these architectures. Asymmetric effects emerge: high dropout reduces memory accuracy by 27 percentage points while reasoning degrades only 1 point, suggesting memory tasks rely on stable representations that dropout disrupts. 84% of models demonstrate memory-biased performance. This provides the first comprehensive MC Dropout benchmark for transformers, revealing dropout robustness is architecture-dependent and uncorrelated with scale. The cognitive profiling framework offers actionable guidance for model selection in uncertainty-aware applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。