A lightweight semantic segmentation model named C2LA-U2-Net, equipped with a cross attention mechanism and a residual refinement module, was proposed to address issues such as the inability to recognize fine features, blurry defect boundaries, and large model parameters in the segmentation of surface defects in polycrystalline solar cells. Firstly, a C2LA module with a cross attention mechanism was designed in the external decoding stage to extract multi-scale spatial features, reduce spatial information loss, and capture long-range dependencies, which enhanced the segmentation performance for small defects. Secondly, a lightweight twostage residual refinement module (D-RRM) was introduced to tackle the issue of blurry prediction boundaries by modeling fine-grained features to improve boundary precision. Finally, Ghost convolutions were incorporated to further reduce model complexity. Experimental results indicated that, compared to the baseline model, the C2LA-U2-Net model achieved improvements of 3.1% in mean pixel accuracy (MPA), 4.49% in mean intersection over union (MIoU), 4.39% in mean recall rate (MRecall), and 4.17% in F1 score. At the same time, the model′s parameters and GFLOPs decreased by 89.77% and 56.68%, respectively, while inference speed increased by 76.97%, demonstrating the effectiveness of the proposed method.