用现有视频模型隐含的3D信息，让机器人更高效地完成复杂操作。
Flex-$π$: A Multi-Stream World-Action Model with Compute Flexibility
- 共享潜空间融合多模态信号，用同一模型处理图像、3D几何和语义
- 在真实双臂操作任务上性能比基线强2-7倍，且推理更快
- 支持灵活配置模态，适合需要少样本泛化的机器人应用
世界-动作模型（WAMs）通过预测未来来提升决策能力，但现有方法仅预测RGB潜在变量，依赖像素重建训练，缺乏对3D几何或物体语义操作的显式信号。我们发现一个意外的免费午餐：相同的冻结视频生成VAE在编码RGB的同时，几乎无损地也编码了3D点云图，无需额外点云训练。这使我们能够在不增加传感器、预训练或推理延迟的前提下，对一个60亿参数的WAM（Flex-π）同时监督其在3D几何和以对象为中心的DINO语义上的表现。所有视觉信号被投影到共享潜空间，并在混合变压器主干中与动作联合去噪；每流丢弃结合跨模态强制机制，使单个训练检查点可运行任意子集的信号流，从快速仅动作模式到完整联合生成。结果是具备极强示范效率且泛化能力强的策略，在分布内和分布外的灵巧、精准真实双臂操作任务上，性能优于最强基线2-7倍，同时运行速度超过π₀.₅。项目主页：https://flex-pi.github.io/
原文摘要 · Abstract (English)
World-action models (WAMs) predict the future to act better, but nearly all of them predict only RGB latents, trained purely for pixel reconstruction, with no explicit signal for the 3D geometry or object semantics manipulation needs. We find a surprising free lunch: the same frozen video-generation VAE that encodes RGB also encodes 3D pointmaps almost losslessly, with no pointmap-specific training at all. This lets us supervise Flex-$π$, a 6B-parameter WAM, on 3D geometry and object-centric DINO semantics alongside RGB, at no cost in new sensors, new pre-training, or inference latency. Every visual signal is projected into this shared latent space and denoised jointly with actions inside a Mixture-of-Transformers backbone; per-stream dropout with cross-modality forcing then lets a single trained checkpoint run on any subset of these streams, from a fast action-only mode to full joint generation. The result is a policy that is exceptionally demonstration-efficient and generalizes well, beating the strongest baselines by up to 2-7$\times$ on dexterous, precise, real-world bimanual manipulation tasks both in and out of distribution, all while running faster than $π_{0.5}$. Our project website: https://flex-pi.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。