To address the decline in detection accuracy caused by complex water surface backgrounds such as wave interference, light reflection, and object occlusion, an improved YOLOv11n-based water surface garbage detection algorithm is proposed. First, the Bottleneck structure in C3k2 is replaced with the FCM module to construct the C3k2-FCM module, mitigating the loss of spatial position information during downsampling. Second, a FPSC-SCSA module is designed by introducing shared convolutional layers with different dilation rates and the SCSA mechanism to replace the SPPF module, enhancing the model′s focus on key regions and reducing the loss of crucial information. Third, a BIFPN-V2S module is developed by replacing the neck aggregation network with a multi-scale feature fusion network embedded with the SDI module from U-Net V2, expanding the receptive field and strengthening global contextual interactions. Finally, an SGSV-IoU loss function is designed to improve boundary shape and detail representation, enhancing the localization of irregular floating objects. Experimental results on a self-built water surface garbage dataset show that, compared with the original YOLOv11n model, the improved algorithm increases mAP50 by 2.8%, while reducing parameters and computation by 0.81 M and 0.1 GFLOPs, respectively, demonstrating the effectiveness of the proposed method.