Road damage increases the likelihood of traffic accidents, posing a serious threat to traffic safety. Therefore, real-time monitoring of road conditions is crucial for ensuring road safety and effectively managing infrastructure. To address the issues of insufficient detection accuracy and small target detection challenges in existing road defect detection methods, this paper proposes an improved RT-DETR-based road defect detection algorithm. First, partial convolution (PConv) is introduced to reconstruct the RT-DETR backbone network, effectively reducing computational overhead. Second, a triplet attention mechanism is integrated into the backbone network to enhance the model′s sensitivity to multi-dimensional features, enabling more precise capture of image details. Next, a BiFPN-based feature pyramid network is employed to optimize the CCFM feature fusion module, and S2 features are introduced to improve the detection performance of small targets. Finally, the DySample upsampling operator is utilized to capture more local details and semantic information, further enhancing the model′s ability to detect small targets. Experimental results show that the improved algorithm achieves a 3.6% increase in mAP@50 on the RDD2022 dataset compared to the original RT-DETR model, with a 12.5% reduction in the number of parameters and a detection speed of 66 fps. Compared with other object detection algorithms, the improved algorithm demonstrates significant advantages in both detection accuracy and speed, making it more suitable for practical applications in road defect detection.