Aiming at the problem of low defect detection accuracy caused by complex steel surface background in industrial production, this paper proposes a steel surface defect detection algorithm based on improved YOLOX. First, the Swin Transformer module is introduced to capture the global context information of the surface area of the defective steel and extract more differentiated features. Secondly, the weighted bidirectional feature pyramid network (BiFPN) is used to facilitate cross-scale feature fusion. Finally, we improved the original target localization loss function and established a CIoU loss function that fuses the center position of the bounding box to achieve high-precision localization of the target frame. Experiments showed that the mAP of our algorithm on the NEU-DET dataset is 80.7%, which is 6.2% higher than the original YOLOX-S network, and it is also significantly higher than some other mainstream algorithms, with high accuracy and practicality.