Wafer defect detection is a critical step in ensuring product quality within semiconductor manufacturing processes. Analyzing the distribution areas and presentation patterns of defects enables precise tracing of weak links in production processes, providing essential evidence for optimizing manufacturing technologies. To address the demands for high precision, real-time processing, and lightweight solutions in identifying hybrid wafer defects under complex scenarios, this paper proposes a lightweight wafer map defect detection algorithm based on an improved YOLOv8s-seg. The algorithm first introduces a context-guided mechanism within the backbone feature extraction network to optimize the C2f module. It then proposes a shift-enhanced multi-branch & scale fusion feature pyramid network (SEMF-FPN) to enhance the model′s feature fusion capabilities. Finally, it incorporates lightweight asymmetric detector head concepts to optimize the head network. Experimental results demonstrate that the improved model achieves mean average precision (mAP@0.5:0.95) of 90.9% for bounding boxes and 84.5% for masks, with a detection speed of 333.3 fps. It significantly reduces parameters and computational load by 54.84% and 37.18%, respectively, while maintaining accuracy loss below 0.5%. This lightweight, high-precision and fast-detection model meets industrial production requirements and holds considerable engineering application value.