Due to the large neural network structure, vast number of parameters, complex environment, and poor performance of hardware equipment in actual engineering inspection sites, the real-time detection rate of defects is slow and the accuracy is low. This study combines the lightweight concept of Depthwise Separable Convolution from MobileNet with the ECA attention mechanism module, as well as the feature extraction model of the U-Net network, to propose a photovoltaic panel defect detection method based on an improved U-Net network model. At the same time, according to the characteristics of photovoltaic cell defects, suitable activation functions were selected and the cross-entropy loss function was improved. Experimental results show that the improved U-Net algorithm not only reduced the number of parameters by 36% compared to the original algorithm, but also achieved a detection accuracy of 97.05% for defects such as cracks and black spots, demonstrating better performance in segmenting surface defects of photovoltaic cells than traditional networks.