将生成式语言模型转为双向编码器，提升多模态表征能力。
BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs
- 通过掩码预训练+权重融合，实现生成模型向双向编码器转化
- 在文本、视觉、音频任务上均超越现有方法，最高提升3.2个点
- 开源通用方案，适合想改造大模型的开发者使用
将因果生成式语言模型转化为双向编码器，可作为BERT类架构的有力替代。然而当前方法仍存在训练目标不统一、大规模下灾难性遗忘、难以整合专用生成模型等问题。本文基于Gemma3和Qwen3系列进行系统消融实验，发现关键在于常被忽略的先验掩码阶段。为在无原始预训练数据条件下扩展，提出线性权重合并与轻量多领域数据混合相结合的双策略，有效缓解遗忘问题。进一步通过与专用因果模型融合，无缝迁移跨模态与领域专长。该开源方案适用于任意因果解码器模型，构建出五款名为BidirLM的编码器，在文本、视觉和音频表征基准上全面超越现有方法。
原文摘要 · Abstract (English)
Transforming causal generative language models into bidirectional encoders offers a powerful alternative to BERT-style architectures. However, current approaches remain limited: they lack consensus on optimal training objectives, suffer from catastrophic forgetting at scale, and fail to flexibly integrate the vast ecosystem of specialized generative models. In this work, through systematic ablations on the Gemma3 and Qwen3 families, we identify the key factors driving successful adaptation, highlighting the critical role of an often-omitted prior masking phase. To scale this process without original pre-training data, we introduce a dual strategy combining linear weight merging with a lightweight multi-domain data mixture that mitigates catastrophic forgetting. Finally, we augment our encoders by merging them with specialized causal models, seamlessly transferring modality- and domain-specific capabilities. This open-source recipe, designed for any causal decoder LLM, yields BidirLM, a family of five encoders that outperform alternatives on text, vision, and audio representation benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。