让视频世界中的角色能实时回应用户互动，一键触发挥手、点头等自然反应。
HelloWorld: Enabling Socially Interactive Characters in Video World Models
- 用自蒸馏生成带交互与镜头运动的数据，训练模型理解相机姿态。
- 按钮按下时，通过调节注意力掩码精准定位角色响应时间窗口。
- 构建400样本基准测试，评估社交互动质量与画面表现的平衡性。
尽管视频世界模型取得了显著进展，但用户与虚拟角色之间的社交互动仍无法支持。为此，我们提出HelloWorld，一种可实现与世界中角色社交互动的视频世界模型。用户只需轻按一下按钮，屏幕角色即可面向摄像头做出回应，如转身、挥手、点头或说一句简短问候。为使互动更自然，我们提出自蒸馏流程，将模型在自身生成的数据上进行微调。每段合成视频均包含社交互动与镜头运动，使模型在不降低交互质量的前提下学习相机姿态条件。推理阶段，我们引入无训练模块，用于判断互动发生时机。按钮按下后，该模块调节DiT的交叉注意力掩码，使交互提示仅关注按键窗口内的帧，实现响应的时间定位。我们进一步构建HelloWorldBench，一个包含400个样本的基准测试，涵盖三种社交互动指标和三种传统指标。实验表明，HelloWorld在交互质量上优于多种基线模型，同时保持最先进的图像美学与相机姿态跟随能力。
原文摘要 · Abstract (English)
Despite the remarkable recent progress of video world models, social interaction between users and the characters within these worlds remains unsupported. To fill this gap, we present HelloWorld, a video world model that enables social interaction with in-world characters. With a single button press, users can prompt the on-screen character to respond toward the camera, e.g., turning to the viewer, waving, nodding, or speaking a short greeting. To make these interactions natural, we propose a self-distillation pipeline that finetunes the video generation model on data synthesized by itself. Each synthesized clip contains both social interactions and camera motion, allowing the model to learn camera-pose conditioning without degrading interaction quality. At inference, we further introduce a training-free module that determines when the interaction occurs. Upon a button press, it modulates the cross-attention masks of the DiT so that the interaction-related text prompt attends only to the frames within the press window, temporally localizing the character's response. We further build HelloWorldBench, a 400-sample benchmark with three social interaction metrics alongside three conventional metrics, for evaluation. Experiments demonstrate that HelloWorld surpasses a variety of baselines in interaction quality, while maintaining state-of-the-art picture aesthetics and camera-pose following. Project page: https://github.com/AlayaLab/HelloWorld
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。