用热扩散原理生成更准的遥感图像令牌，提升大模型理解能力
HeatTok: Enhancing Remote Sensing Image Understanding via Thermodiffusion-based Tokenization
- 基于热扩散原理自适应合并同质区域，生成符合物体形状的不规则令牌
- 在VRSBench和EarthVQA上以合理令牌数量达到领先性能
- 专为不规则地理对象设计位置编码，适合遥感与多模态大模型研究者
当前多模态大模型的视觉令牌化方法主要依赖块划分，导致遥感图像中地理对象因轮廓不规则而出现语义混杂和物体碎片化。现有自适应方法难以精准提取对象级令牌，且缺乏针对不规则区域的专用几何位置编码。本文提出HeatTok，一种基于热扩散聚合的语义感知令牌化方法。受热传导物理原理启发，HeatTok自适应合并相邻同质区域，生成语义独立、与物体对齐的不规则令牌。为使多模态大模型感知这些不规则形状，设计了高斯多模态旋转位置编码（G-MRoPE），通过二维高斯建模令牌空间分布，并显式注入中心、尺度和方向信息。在VRSBench和EarthVQA数据集上的大量实验表明，HeatTok有效保持了对象级语义完整性，在合理令牌预算下达到最先进性能。代码已开源：https://github.com/YingyingYan1/HeatTok。
原文摘要 · Abstract (English)
Current visual tokenizers in Multimodal Large Language Models (MLLMs) predominantly rely on patch-based partitioning, which causes severe semantic mixture and object fragmentation in remote sensing imagery due to the irregular contours of geo-objects. Moreover, existing adaptive methods struggle to extract precise object-level tokens and lack dedicated geometric positional encodings for irregular regions. In this paper, we propose HeatTok, a semantic-aware tokenizer driven by thermodiffusion aggregation. Inspired by the physical principles of heat conduction, HeatTok adaptively merges adjacent homogeneous regions to generate semantically independent, object-aligned irregular tokens. To enable MLLMs to perceive these irregular shapes, we design the Gaussian Multimodal Rotary Positional Embedding (G-MRoPE), which models token spatial distributions via 2D Gaussians and explicitly injects center, scale, and orientation cues. Extensive evaluations on the VRSBench and EarthVQA datasets demonstrate that HeatTok effectively preserves object-level semantic integrity and achieves state-of-the-art performance under a reasonable token budget. The code is available: https://github.com/YingyingYan1/HeatTok.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。