提出几何引导的负载均衡方法，解决视觉语言混合专家模型中图文负载不均问题。
Relax Within, Balance Across: Geometry-Guided Load Balancing for Vision-Language Mixture-of-Experts
- 基于图文分块结构设计分离路由机制，实现模态内与模态间负载均衡。
- 在不同图像分辨率下，负载不平衡度降低超五倍，且平均负载显著下降。
- 适用于多模态任务中对负载稳定性要求高的场景，如跨分辨率推理。
视觉语言混合专家（Vision-Language MoE）批次中包含不同数量的图像和文本标记。图像分辨率、图像数量、分块方式及提示长度均会改变标记构成。标准的标记级切换辅助损失（Std-Aux）仅平衡混合负载，导致大图像与文本负载误差在特定混合比例下相互抵消。在主模型上，相同训练的路由器在不同图像分辨率下表现出超过五倍的负载不平衡变化。我们固定图像与文本负载分布，推导出随标记混合变化的精确负载曲线。图像-文本负载差距决定对混合比例的敏感性。物理预处理也会改变条件分布，而固定分布规律排除了此类影响。为设计解决方案，我们分析路由器输入结构：图像与文本占据不同区域，视觉标记按来源图像强聚类。模态边界启发分别建模图像与文本项，图像边界启发每张图像使用一个等权重路由实例。ReBA（Relax Within, Balance Across）实现了上述两个设计选择。在四个拆分骨干网络上，ReBA在所有报告基准输入下均降低负载，同时保持与Std-Aux相当的平均任务准确率。此外，ReBA还降低了测试范围内的平均负载以及分辨率和分块扰动下的最差物理负载。代码已开源：https://github.com/ZiangWu-77/ReBA。
原文摘要 · Abstract (English)
Vision-language MoE batches contain different numbers of image and text tokens. Image resolution, image count, tiling, and prompt length all change this token mix. We call the standard token-level Switch auxiliary loss Std-Aux. Std-Aux balances only the mixed load, so large image and text load errors can cancel at one mix. On our main model, the same trained router shows more than a fivefold change in load imbalance across image resolutions. We hold the image and text load profiles fixed and derive the exact load curve as the token mix varies. The image-text load gap controls sensitivity to the token mix. Physical preprocessing can also change the conditional profiles. The fixed-profile law excludes such changes. To design a remedy, we examine the router input structure. Image and text occupy distinct regions, while visual tokens group strongly by source image. The modality boundary motivates separate image and text terms. The image boundary motivates one equal-weight routing instance per image. ReBA, or Relax Within, Balance Across, implements both choices. Across four split backbones, ReBA lowers load on every reported benchmark input while keeping mean task accuracy comparable to Std-Aux. ReBA also lowers average load over the tested range and worst physical load under resolution and tiling shifts. Code is available at https://github.com/ZiangWu-77/ReBA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。