Application of Vision-Language Model to Pedestrians Behavior and Scene Understanding in Autonomous Driving

Autonomous driving advances face challenges in understanding pedestrians. Vision-language models, adept at scene understanding and planning, offer promise but require significant computational resources. This paper analyzes effective knowledge distillation of large language model semantic labels to smaller vision networks, enhancing scene representation for autonomous driving decision-making. It discusses deploying large language and vision models on vehicles, focusing on developing specialized models, efficient deployment strategies, and methods to generate actionable signals for vehicle control. The paper introduces a comprehensive pedestrian semantic attributes taxonomy, enabling more intelligent and responsive autonomous vehicles. The Waymo Open Dataset is a resource for pedestrian behavior prediction, featuring diverse real-world scenarios. GPT4-V generates annotations for pedestrians, focusing on actions, behaviors, and unusual situations. The text outlines a method using GPT annotations for autonomous driving, formulating the problem as a multi-label classification, predicting probabilities of semantic labels appearing in GPT outputs. It discusses binary cross-entropy loss, comparing CNN and Vision Transformer backbones for semantic embedding, and introduces CLIP, an influential model for multi-modal learning. The text evaluates ensemble models for text generation, focusing on metrics like BLEU score, and discusses trajectory prediction tasks for autonomous vehicles, using pedestrian behavior signals and latent semantic embedding. The study achieved significant reductions in trajectory errors at 3 seconds compared to a baseline.