Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation

Track4Gen integrates video diffusion loss and point tracking for enhanced spatial supervision, addressing appearance drift in video generation. By unifying video generation and point tracking, it demonstrates effective reduction of appearance drift, resulting in temporally stable, visually coherent videos. The framework upgrades existing video generators by adding a correspondence tracking loss, improving appearance constancy in diffusion-based videos. Evaluations show significant enhancement in video generation quality compared to conventional metrics and recent benchmarks. Track4Gen uses Stable Video Diffusion, a latent image-to-video model, and evaluates its performance on the VBench dataset, showing superior results. The method leverages video models to enhance spatial awareness in intermediate features, boosting performance in correspondence tracking and video generation.