In road damage detection tasks using UAV aerial images, existing algorithms face challenges including high computational complexity, false negatives, and false positives in complex backgrounds. To address these problems, we propose a lightweight road damage detection model, DFS-YOLO. First, we introduce the C2f-DWR module, which employs a parallel structure with dilated convolutions of multiple dilation rates to expand the model′s receptive field and enhance the utilization of high-level semantic information. Second, we design a lightweight Faster Hierarchical Scale-based Feature Pyramid Network (FHSFPN) to reduce model complexity while improving feature fusion. Finally, we introduce the ShapeIoU loss function, which focuses on the shape and scale of road damage to improve the model′s robustness. Experimental results demonstrate that DFS-YOLO outperforms YOLOv8s, achieving a 4.6% and 2.1% improvement in mAP50 on the China Drone and UAPD datasets, respectively. Additionally, the model reduces the number of parameters and computational complexity by 39.1% and 20.4%, respectively, achieving a good balance between lightweight design and accuracy. These results highlight its significant potential for practical applications.