微调用户提供的边界框，让视频生成更符合意图且质量更高
Making Video Models Adhere to User Intent with Minor Adjustments
- 通过优化边界框位置使其匹配模型内部注意力图
- 微调后生成视频质量显著提升，控制精度明显增强
- 适合需要精准控制视频生成的创作者和研究者
随着文本到视频扩散模型的快速发展，如何有效控制生成内容成为研究热点。目前常用的方法是使用边界框或布局进行控制，但确保生成结果严格遵循这些控制输入仍具挑战。本文发现，仅对用户提供的边界框进行轻微调整，即可显著提升生成质量和控制一致性。方法核心是优化边界框位置，使其与视频扩散模型内部的注意力图更好地对齐，同时平衡前景与背景的关注度。我们提出一种平滑掩码使边界框位置可微，并采用注意力最大化目标来调整边界框。通过大量实验，包括用户研究，验证了该方法的有效性。代码已公开，以促进社区后续研究。
原文摘要 · Abstract (English)
With the recent drastic advancements in text-to-video diffusion models, controlling their generations has drawn interest. A popular way for control is through bounding boxes or layouts. However, enforcing adherence to these control inputs is still an open problem. In this work, we show that by slightly adjusting user-provided bounding boxes we can improve both the quality of generations and the adherence to the control inputs. This is achieved by simply optimizing the bounding boxes to better align with the internal attention maps of the video diffusion model while carefully balancing the focus on foreground and background. In a sense, we are modifying the bounding boxes to be at places where the model is familiar with. Surprisingly, we find that even with small modifications, the quality of generations can vary significantly. To do so, we propose a smooth mask to make the bounding box position differentiable and an attention-maximization objective that we use to alter the bounding boxes. We conduct thorough experiments, including a user study to validate the effectiveness of our method. Our code is made available on the project webpage to foster future research from the community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。