用自适应网格生成多尺度令牌,让神经PDE模型更高效精准。
MeshTok: Efficient Multi-Scale Tokenization for Scalable PDE Transformers

- 基于自适应网格细化生成不均匀多尺度令牌
- 在多个PDE数据集上提升效率与精度平衡
- 适合需要处理复杂多尺度物理问题的研究者
传统分块Transformer对空间进行均匀划分,计算资源平均分配,无法有效捕捉复杂偏微分方程(PDE)的局部特征。为此,我们提出MeshTok,一种受自适应网格细化(AMR)启发的令牌化与序列建模框架。该方法针对具有陡峭梯度、瞬态特征或多重尺度结构的区域进行选择性细化,在固定模拟网格上生成异构的多尺度令牌。这些令牌统一输入Transformer序列,使模型能同时捕获全局上下文与局部细节,无需特殊架构。尽管自适应细化适度增加令牌数量,但将计算资源集中于物理信息丰富的区域,形成实用的归纳偏置而非形式最优保证。在多个PDE族和基准数据集上的实验表明,与均匀网格基线相比,MeshTok始终改善了效率-精度权衡。这表明自适应多尺度令牌化是神经PDE建模中可扩展且通用的设计原则。代码已开源:https://github.com/SCAILab-USTC/MeshTok。
原文摘要 · Abstract (English)
Conventional patchified Transformers operate on uniform spatial partitions, distributing computational effort evenly across the domain irrespective of local features. This inflexible tokenization scheme is inherently limited in its ability to efficiently represent and process solutions to complex PDEs. To address this, we propose MeshTok, an adaptive mesh refinement (AMR)-inspired tokenization and sequence modeling framework. This method selectively refines spatial regions exhibiting sharp gradients, transient features, or multiscale structures, generating a heterogeneous set of multiscale tokens defined on a fixed simulation grid. These tokens are processed within a unified Transformer sequence, enabling the model to simultaneously capture coarse-grained global context and fine-grained local details without requiring specialized architectural components. Although adaptive refinement moderately increases token count, it promotes a more targeted allocation of computational resources to physically informative regions, which we view as a practical inductive bias rather than a formal optimality guarantee. Experimental evaluations across multiple PDE families and benchmark datasets demonstrate that MeshTok consistently improves the efficiency-accuracy trade-off compared to uniform-grid baselines. This suggests adaptive multiscale tokenization as a scalable and generalizable design principle for neural PDE modeling. Code is available at https://github.com/SCAILab-USTC/MeshTok.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。