Multi-Modal Fusion for Road Infrastructure Analysis: An AI-Driven Framework for Heterogeneous Data

Research objectives

• Advance Automated Road Safety Auditing through Vision-Language Models: Transition road infrastructure assessment from rigid, closed-set classifiers to multimodal Vision-Language Models (VLMs), enabling the automated generation of comprehensive and structured records directly from road survey imagery.
• Develop Robust Spatio-Temporal Video Modeling Frameworks: Design deep learning architectures that effectively capture both persistent infrastructure characteristics (e.g., lane configurations and medians) and transient, localized features (e.g., roadside hazards).
• Tackle Extreme Data Imbalance and Imperfect Real-World Annotations: Formulate adaptive optimization strategies and field-level noise-filtering techniques that prioritize rare, safety-critical hazards over common classes while systematically mitigating human labeling inconsistencies without sacrificing usable visual data.
• Integrate Heterogeneous Geospatial Data for Cross-Regional Generalizability: Combine complementary data sources, such as street-level RGB video, 3D LiDAR point clouds, and GIS mapping data, to resolve visual and depth ambiguities, ensuring the developed models generalize reliably across diverse, international road networks.
• Establish Trustworthy AI Systems: Incorporate reliable confidence estimation into multimodal models to enable practical “human-in-the-loop” deployments, allowing the AI to autonomously process high-confidence road segments while systematically escalating ambiguous, high-risk cases for expert human review.