Vision-enhanced multi-modal learning framework for non-destructive pavement damage detection
Yingchao Zhang, Cheng Liu
Abstract
Road infrastructure damage detection is crucial for maintaining transportation safety and efficiency. While current methods like machine learning and computer vision are effective, they have difficulty analyzing and describing complex pavement conditions. This paper presents DamageQwen, a vision-language model framework for non-destructive testing of pavement damage. The framework integrates YOLO v11x for initial damage detection with vision-language models (VLMs) to enhance detection accuracy and reduce reliance on large labeled datasets. By combining object detection capabilities with natural language understanding, DamageQwen enables more comprehensive damage assessment through visual question answering. The framework incorporates CLIP-based repeat defects elimination and supports few-shot learning with just 1–4 shots. Compared to generic VLMs, our approach significantly improves the accuracy of descriptions while greatly reducing the labelling effort. The framework's ability to process both visual and semantic information allows non-experts to effectively detect and assess pavement damage through simple prompts.