约束融合让文本信息更好辅助时间序列预测
Rethinking Multimodal Fusion for Time Series: Text Modalities Need Constrained Fusion
- 用低秩适配器筛选相关文本信息，控制跨模态融合
- 在20000+实验中，约束融合显著优于简单拼接方法
- 无需改动主模型，适合想加文本的时序任务研究者
近期多模态学习进展推动了文本、视觉等辅助模态融入时间序列（TS）预测。然而，现有方法增益有限，常仅在特定数据集有效，或依赖架构特异性设计，难以泛化。本文发现，采用简单融合策略（如直接相加或拼接）的多模态模型往往表现不如单模态时序模型，原因在于未受控地引入无关信息。基于此，我们探索多种受控融合方法，发现其始终优于朴素融合。进一步提出可控融合适配器（CFA），一种无需修改时序主干网络的即插即用模块，仅融合与时序动态对齐的文本信息。CFA通过低秩适配器过滤无关文本，在融合前进行特征筛选。我们在多个数据集和时序/文本模型上进行了超过20,000次实验，验证了受控融合的有效性。代码已开源：https://github.com/seunghan96/cfa。
原文摘要 · Abstract (English)
Recent advances in multimodal learning have motivated the integration of auxiliary modalities such as text or vision into time series (TS) forecasting. However, most existing methods provide limited gains, often improving performance only in specific datasets or relying on architecture-specific designs that limit generalization. In this paper, we show that multimodal models with naive fusion strategies (e.g., simple addition or concatenation) often underperform unimodal TS models, which we attribute to the uncontrolled integration of auxiliary modalities which may introduce irrelevant information. Motivated by this observation, we explore various constrained fusion methods designed to control such integration and find that they consistently outperform naive fusion methods. Furthermore, we propose Controlled Fusion Adapter (CFA), a simple plug-in method that enables controlled cross-modal interactions without modifying the TS backbone, integrating only relevant textual information aligned with TS dynamics. CFA employs low rank adapters to filter irrelevant textual information before fusing it into temporal representations. We conduct over 20K experiments across various datasets and TS/text models, demonstrating the effectiveness of the constrained fusion methods. Code is available at: https://github.com/seunghan96/cfa.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。