语言、视觉与动作表征在智能体中存在深层对齐,支持跨模态语义共享。
Alignment among Language, Vision and Action Representations
- 用婴儿AI平台生成仅依赖感知-动作控制的语言嵌入
- 动作表征与解码器型语言模型对齐度达0.70-0.73(precision@15)
- 揭示了具身智能系统中跨域迁移的潜力
认知科学与人工智能的核心问题之一是:语言、视觉与动作等不同学习模态是否产生独立或共享的内部表征。传统观点认为,基于不同数据训练的模型会形成专用且不可转移的表征。然而,近期证据表明,即使优化目标各异,模型也可能发展出相似的表征几何结构。我们通过在婴儿AI平台上使用行为克隆训练一个基于Transformer的智能体,使其根据自然语言指令执行目标导向行为,从而生成仅由感知-动作控制需求塑造的动作接地语言嵌入。随后,我们将这些表征与当前主流大语言模型(LLaMA、Qwen、DeepSeek、BERT)及视觉-语言模型(CLIP、BLIP)的表征进行对比。尽管训练数据、模态和目标差异显著,我们仍观察到稳健的跨模态对齐:动作表征与解码器型语言模型及BLIP表现出强对齐(precision@15: 0.70–0.73),接近语言模型之间的对齐水平;而与CLIP和BERT的对齐则明显较弱。结果表明,语言、视觉与动作表征会收敛至部分共享的语义结构,支持模态无关的语义组织,并凸显具身智能系统中跨领域迁移的可能。
原文摘要 · Abstract (English)
A fundamental question in cognitive science and AI concerns whether different learning modalities: language, vision, and action, give rise to distinct or shared internal representations. Traditional views assume that models trained on different data types develop specialized, non-transferable representations. However, recent evidence suggests unexpected convergence: models optimized for distinct tasks may develop similar representational geometries. We investigate whether this convergence extends to embodied action learning by training a transformer-based agent to execute goal-directed behaviors in response to natural language instructions. Using behavioral cloning on the BabyAI platform, we generated action-grounded language embeddings shaped exclusively by sensorimotor control requirements. We then compared these representations with those extracted from state-of-the-art large language models (LLaMA, Qwen, DeepSeek, BERT) and vision-language models (CLIP, BLIP). Despite substantial differences in training data, modality, and objectives, we observed robust cross-modal alignment. Action representations aligned strongly with decoder-only language models and BLIP (precision@15: 0.70-0.73), approaching the alignment observed among language models themselves. Alignment with CLIP and BERT was significantly weaker. These findings indicate that linguistic, visual, and action representations converge toward partially shared semantic structures, supporting modality-independent semantic organization and highlighting potential for cross-domain transfer in embodied AI systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。