构建空中追踪的结构化规划与多模态仿真系统,支持真实感场景模拟。
CosFly: Plan in the Matrix, Fly in the World

- 将3D环境转化为障碍物结构图,分7步生成轨迹并映射到多模态数据
- 提供250条验证轨迹、约10万张带6-DOF姿态标注的图像
- 适合研究无人机导航、多模态感知及空地协同的团队使用
我们提出CosFly,一个面向空中目标追踪的盒式结构规划与多模态仿真流水线,以及大规模无人机数据集CosFly-Track,涵盖城市中心、高速公路、乡村、森林和沿海小镇等多种动态环境。当前在CARLA中的实现包含7步模块化流程:从3D地图导出、网格简化、行人与无人机轨迹规划、多模态渲染(含RGB图像、高精度深度图、语义分割掩码)及6-DOF姿态标注,到质量检查与师生生成自然语言导航指令。支持可配置固定视场角(每条轨迹设定一次,全程保持不变),通过相机内参调整模拟不同焦距。对比分析两种轨迹规划范式:传统两阶段(前端候选生成+后端优化)与直接梯度优化法(单目标优化多个追踪约束)。公开发布的CosFly-Track包含250条验证轨迹与约10万张带完整6-DOF无人机位姿标注(位置x,y,z与朝向yaw,pitch,roll)的图像。该流水线与数据集为跨环境空地协同研究提供可扩展基础,支持动态目标追踪、无人机导航与多模态感知。
原文摘要 · Abstract (English)
We present CosFly, a box-structured planning and multimodal simulation pipeline for aerial tracking, together with CosFly-Track, a large-scale UAV dataset for dynamic target tracking across diverse environments including urban centers, highways, rural landscapes, forests, and coastal towns. In our current implementation on CARLA, CosFly provides a modular 7-step construction pipeline that converts complex 3D worlds into structured obstacle representations for planning, then projects the resulting trajectories back into multi-modal sensor data -- including RGB images, high-precision depth maps, and semantic segmentation masks -- paired with natural language navigation instructions. A key feature is the support for configurable fixed-FOV zoom levels (one FOV setting drawn per trajectory and held constant throughout), enabling simulation of various focal lengths through camera-intrinsic adjustments. The pipeline covers the complete workflow from 3D map export through grid simplification, pedestrian and drone trajectory planning, multi-modal rendering with 6-DOF pose annotations, quality inspection, and teacher-student caption generation. We analyze two trajectory-planning paradigms for aerial target tracking: a conventional two-stage pipeline with front-end candidate generation and backend refinement, and a direct gradient-based formulation that optimizes multiple tracking constraints in a single objective. The public CosFly-Track release contains 250 validated trajectories and approximately 100,000 rendered images with complete 6-DOF drone pose annotations (position x, y, z and orientation yaw, pitch, roll). Together, the pipeline and dataset establish a scalable foundation for aerial-ground collaborative research, supporting dynamic target tracking, UAV navigation, and multi-modal perception across diverse environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。