Addressing the issues of missed and false detections of pallets in real factory environments, often caused by factors such as insufficient lighting and numerous obstacles, a tray detection method based on an improved YOLOv8n is proposed. Firstly, the Bi-Level Routing Attention (BRA) sparse attention module combined with Transformer is incorporated into the backbone network feature extraction phase of the YOLOv8n model, to reduce the interference from obstacle occlusion on pallet detection. Secondly, the Shape-IoU loss function is introduced, further enhancing the model′s ability to recognize pallets in conditions of insufficient lighting and severe background interference. Finally, the feature fusion network of YOLOv8n is reconstructed using the GSConv-based Slim-neck structure, achieving a lightweight neck network. Experimental results indicate that the improved algorithm achieves a mean Average Precision (mAP) of 89.6% on the test set, representing a 2.8% improvement compared to the original model. The missed detection rate and false detection rate decrease by 2% and 2.2%, respectively. This effectively mitigates the problems of missed and false detections of pallets in situations of insufficient lighting and obstacle occlusion. Additionally, with a detection frame rate of 312.5 fps, the method enables rapid and accurate pallet detection and recognition, making it suitable for deployment on smart forklifts to enhance operational efficiency and elevate warehouse intelligence levels.