arXiv:2603.17555cs.CVcs.AI2026-03

让4K图像生成视频保持全局结构一致,不丢失细节。

FrescoDiffusion: 4K Image-to-Video with Prior-Regularized Tiled Diffusion

  • 用预计算的低分辨率视频潜变量作全局参考,指导高分辨率分块生成。
  • 每步通过加权最小二乘融合噪声预测,提升时空一致性且保留局部细节。
  • 支持区域级运动控制,可调节创造性和一致性平衡,适合壁画动画生成。

基于扩散模型的图像到视频(I2V)生成虽日益高效,但难以扩展至超高清输入(如4K)。在模型原生分辨率下生成易丢失细粒度结构,而高分辨率分块去噪虽保留局部细节,却破坏全局布局一致性。这一问题在壁画动画场景中尤为严重:大型艺术作品包含多个角色、物体及语义各异的子场景,需保持长时间空间连贯性。本文提出FrescoDiffusion,一种无需训练的大型复杂图像生成连贯视频的方法。核心思想是将分块去噪与预计算的潜在先验结合:先以模型底层分辨率生成低分辨率视频,再上采样其潜变量轨迹,获得捕捉长程时空结构的全局参考。对于4K生成,每步计算分块噪声预测,并通过在模型输出空间最小化单一加权最小二乘目标,将其与该参考融合。该目标整合标准分块合并准则与正则化项,得出闭式融合更新,强化全局一致性同时保留细节。此外,引入空间正则化变量,实现对运动区域的局部控制。在VBench-I2V和自建的fresco I2V数据集上的实验表明,相比分块基线,本方法在全局一致性与保真度上均有提升,且计算高效。正则化机制支持显式调控创造性与一致性之间的权衡。

原文摘要 · Abstract (English)

Diffusion-based image-to-video (I2V) models are increasingly effective, yet they struggle to scale to ultra-high-resolution inputs (e.g., 4K). Generating videos at the model's native resolution often loses fine-grained structure, whereas high-resolution tiled denoising preserves local detail but breaks global layout consistency. This failure mode is particularly severe in the fresco animation setting: monumental artworks containing many distinct characters, objects, and semantically different sub-scenes that must remain spatially coherent over time. We introduce FrescoDiffusion, a training-free method for coherent large-format I2V generation from a single complex image. The key idea is to augment tiled denoising with a precomputed latent prior: we first generate a low-resolution video at the underlying model resolution and upsample its latent trajectory to obtain a global reference that captures long-range temporal and spatial structure. For 4K generation, we compute per-tile noise predictions and fuse them with this reference at every diffusion timestep by minimizing a single weighted least-squares objective in model-output space. The objective combines a standard tile-merging criterion with our regularization term, yielding a closed-form fusion update that strengthens global coherence while retaining fine detail. We additionally provide a spatial regularization variable that enables region-level control over where motion is allowed. Experiments on the VBench-I2V dataset and our proposed fresco I2V dataset show improved global consistency and fidelity over tiled baselines, while being computationally efficient. Our regularization enables explicit controllability of the trade-off between creativity and consistency.

视频生成扩散模型4K生成结构一致性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。