提出新指标TopoLip，发现注意力模型比卷积模型更平滑也更鲁棒。
Is Smoothness the Key to Robustness? A Comparison of Attention and Convolution Models Using a Novel Metric
- 用层间分析结合拓扑与Lipschitz连续性定义新指标TopoLip。
- 实证和理论均表明注意力模型变换更平滑，鲁棒性更强。
- 适合关注模型结构与鲁棒性关系的研究者参考。
鲁棒性是机器学习模型的关键特性。现有评估方法或缺乏理论普适性，或过度依赖经验评估，难以揭示影响鲁棒性的结构性因素。此外，理论分析无法直接比较不同模型。为此，我们提出基于层间分析的指标TopoLip，融合拓扑数据分析与Lipschitz连续性，构建统一的理论与实证鲁棒性评估框架，揭示模型参数对鲁棒性的影响。通过TopoLip，我们证明注意力模型通常具有更平滑的特征变换和更强的鲁棒性，该结论在理论分析与对抗任务中得到验证。研究建立了架构设计、鲁棒性与拓扑属性之间的关联。
原文摘要 · Abstract (English)
Robustness is a critical aspect of machine learning models. Existing robustness evaluation approaches often lack theoretical generality or rely heavily on empirical assessments, limiting insights into the structural factors contributing to robustness. Moreover, theoretical robustness analysis is not applicable for direct comparisons between models. To address these challenges, we propose $\textit{TopoLip}$, a metric based on layer-wise analysis that bridges topological data analysis and Lipschitz continuity for robustness evaluation. TopoLip provides a unified framework for both theoretical and empirical robustness comparisons across different architectures or configurations, and it reveals how model parameters influence the robustness of models. Using TopoLip, we demonstrate that attention-based models typically exhibit smoother transformations and greater robustness compared to convolution-based models, as validated through theoretical analysis and adversarial tasks. Our findings establish a connection between architectural design, robustness, and topological properties.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。