Aiming at the three problems of complex background, inconsistent target size and small proportion of defective areas to be inspected in aerial insulator images taken by UAVs during transmission line inspection, a lightweight insulator defect detection algorithm, MHD-YOLO, is proposed. Firstly, a feature extraction network MAFNet is introduced into the backbone network of YOLOv8, and hybrid convolution is used to enhance the feature extraction capability of the network under complex background. Second, a feature fusion network, HS-FPN, is used to realize feature fusion at different scales, and combined with a lightweight dynamic up-sampling method, DySample, to improve the quality and efficiency of up-sampling. Then, a lightweight detection head CSH is designed, which significantly reduces the number of parameters in the detection layer and the computation amount by using the shared convolution method. Finally, the NWD loss function is introduced to improve the localization accuracy of the model for small targets. The experimental results demonstrate that the MHD-YOLO target detection algorithm reduces the number of parameters by 43.8% compared with YOLOv8, and improves the detection accuracy by 5.1% on the insulator defect detection dataset. The improved algorithm is significantly more effective in detecting insulator defects, and the model complexity is greatly reduced, providing a more effective method for deployment on embedded devices.