用文字实时控制3D高斯场景中的物理动画，无需训练
PhysTalk: Language-driven Real-time Physics in 3D Gaussian Scenes
- 用大语言模型生成代码，直接操控3D高斯参数实现物理交互
- 支持多材质物体碰撞，4D动画响应实时，延迟低于100ms
- 无需训练和网格提取，适合创意设计与交互式模拟场景
真实视觉模拟应用广泛，但需大量计算、渲染和专业动画知识。基于文本的开放词汇视觉特效生成有望释放巨大创作潜力，但现有方法缺乏物理真实性和高效语言接口，依赖缓慢的离线优化。相比之下，PhysTalk以3D高斯溅射（3DGS）场景为输入，将任意用户提示实时转化为物理驱动的4D交互动画。大型语言模型（LLM）生成可执行代码，通过轻量级代理和粒子动力学直接修改3DGS参数。值得注意的是，PhysTalk是首个在不依赖耗时网格提取的情况下，将3DGS与物理模拟器直接耦合的框架。在保持开放词汇的同时，该设计实现了对任意多材质物体的碰撞感知、物理驱动的交互式3D高斯动画。此外，PhysTalk无需训练且计算轻量，使4D动画广泛可及，推动工作流从“渲染等待”转向与现代物理驱动管线的互动对话。
原文摘要 · Abstract (English)
Realistic visual simulations are omnipresent, yet their creation requires computing time, rendering, and expert animation knowledge. Open-vocabulary visual effects generation from text inputs emerges as a promising solution that can unlock immense creative potential. However, current pipelines lack both physical realism and effective language interfaces, requiring slow offline optimization. In contrast, PhysTalk takes a 3D Gaussian Splatting (3DGS) scene as input and translates arbitrary user prompts into real time, physics based, interactive 4D animations. A large language model (LLM) generates executable code that directly modifies 3DGS parameters through lightweight proxies and particle dynamics. Notably, PhysTalk is the first framework to couple 3DGS directly with a physics simulator without relying on time consuming mesh extraction. While remaining open vocabulary, this design enables interactive 3D Gaussian animation via collision aware, physics based manipulation of arbitrary, multi material objects. Finally, PhysTalk is train-free and computationally lightweight: this makes 4D animation broadly accessible and shifts these workflows from a "render and wait" paradigm toward an interactive dialogue with a modern, physics-informed pipeline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。