Aiming at the problem that YOLOv4 has a huge backbone network and a large number of parameters, it cannot meet real-time requirements when applied to insulator defect detection. A lightweight YOLOv4 detection model is proposed. First, GhostNet with ECA integrated components is introduced as the feature extraction network, which greatly reduces the model parameters and speeds up the model inference while ensuring the feature extraction capability. Secondly, the K-means++ clustering algorithm is used to determine the initial anchor frame size to adapt to the size of the insulator defect and improve the accuracy of defect location. Finally, on the basis of the cross-entropy loss function, the Quality Focal Loss is introduced to improve the loss function to further improve the model detection performance. Experimental results show that compared with the original YOLOv4, the improved lightweight YOLOv4 has a reduced model size of 62.47%, Frames Per Second increased by 68.83%, and the accuracy of insulator defect detection has increased by 1.07%, significantly improving the detection speed. At the same time, the detection accuracy of the algorithm is guaranteed, and it performs outstandingly in small targets and complex backgrounds.