arXiv:2412.14158cs.CVcs.AI2024-12CVPR被引 19

让AI视频生成可控制镜头运动与光学效果,实现电影级画面

AKiRa: Augmentation Kit on Rays for optical video generation

  • 通过光线增强框架训练相机适配器,精细调控镜头参数
  • 支持变焦、鱼眼、虚化等复杂光学效果,生成更富电影感视频
  • 适合影视创作、动画设计等需要镜头语言控制的场景

文本条件视频扩散模型虽显著提升了视频质量,但在相机动态运动、变焦、畸变和焦点变化等方面的用户控制能力仍有限。这些光学与运动特性对影视内容的情感引导、焦点吸引和氛围营造至关重要。本文提出AKiRa(Augmentation Kit on Rays),一种基于现有视频生成主干网络的新型增强框架,通过复杂相机模型构建并训练相机适配器,实现对相机运动及焦距、畸变、光圈等光学参数的细粒度控制,可生成变焦、鱼眼、背景虚化等电影级视觉效果。大量实验表明,AKiRa在组合与编排光学特性方面表现卓越,优于所有当前最优方法。该工作为可控且光学增强的视频生成树立了新标杆,推动未来光学视频生成技术发展。

原文摘要 · Abstract (English)

Recent advances in text-conditioned video diffusion have greatly improved video quality. However, these methods offer limited or sometimes no control to users on camera aspects, including dynamic camera motion, zoom, distorted lens and focus shifts. These motion and optical aspects are crucial for adding controllability and cinematic elements to generation frameworks, ultimately resulting in visual content that draws focus, enhances mood, and guides emotions according to filmmakers' controls. In this paper, we aim to close the gap between controllable video generation and camera optics. To achieve this, we propose AKiRa (Augmentation Kit on Rays), a novel augmentation framework that builds and trains a camera adapter with a complex camera model over an existing video generation backbone. It enables fine-tuned control over camera motion as well as complex optical parameters (focal length, distortion, aperture) to achieve cinematic effects such as zoom, fisheye effect, and bokeh. Extensive experiments demonstrate AKiRa's effectiveness in combining and composing camera optics while outperforming all state-of-the-art methods. This work sets a new landmark in controlled and optically enhanced video generation, paving the way for future optical video generation methods.

视频生成扩散模型镜头控制电影效果

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。