让Transformer自动选择最适合的几何空间,提升复杂关系推理能力。
CAT: Curvature-Adaptive Transformers for Geometry-Aware Learning
- 通过轻量门控机制动态分配每个令牌到三种几何注意力分支
- 在知识图谱任务中相较固定几何模型提升10%的准确率,参数仅增加5%
- 适合需要处理混合几何结构的多模态与语言任务
Transformer在多个领域表现优异,但其注意力机制隐式假设欧几里得几何,限制了对非欧几里得结构数据的效果。尽管近期在双曲和球面空间的扩展对层次和循环模式有效,但需预先指定单一几何,灵活性不足。我们提出曲率自适应Transformer(CAT),通过轻量可微门控机制,动态学习每个令牌在三种几何注意力分支间的路由。不同于固定几何方法,CAT实现几何特性的自适应分配,将令牌路由至与其局部关系结构匹配的曲率空间。路由网络提供可解释的曲率偏好,各分支采用针对特定流形优化的操作。在知识图谱补全基准(FB15k-237、WN18RR)上,CAT相较固定几何基线在MRR和Hits@10上提升约10%,参数仅增加5%,推理时间相近。结果表明,学习到的几何自适应优于任何单一固定几何,为语言、视觉及多模态领域中的混合几何架构提供了可扩展且可解释的基础。
原文摘要 · Abstract (English)
Transformers achieve strong performance across diverse domains but implicitly assume Euclidean geometry in their attention mechanisms, limiting their effectiveness on data with non-Euclidean structure. While recent extensions to hyperbolic and spherical spaces show promise for hierarchical and cyclical patterns, respectively, they require committing to a single geometry a priori, reducing flexibility when data exhibits mixed geometric properties. We introduce the Curvature-Adaptive Transformer (CAT), a novel architecture that dynamically learns per-token routing across three geometric attention branches through a lightweight, differentiable gating mechanism. Unlike fixed-geometry approaches, CAT enables adaptive geometric specialization, routing tokens to the appropriate curvature based on their local relational structure. The routing network provides interpretable curvature preferences while each branch employs geometry-specific operations optimized for its respective manifold. On knowledge graph completion benchmarks (FB15k-237, WN18RR), CAT achieves approximately 10% improvements in MRR and Hits@10 over fixed-geometry baselines with minimal overhead (5% parameter increase, comparable inference time). These results demonstrate that learned geometric adaptation outperforms any single fixed geometry for complex relational reasoning, establishing CAT as a scalable and interpretable foundation for mixture-of-geometry architectures across language, vision, and multimodal domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。