With the acceleration of urbanization, the continuous increase in domestic waste has posed a severe challenge to the ecological environment. Therefore, intelligent sorting technology based on target detection has become a key solution. Aiming at the problems of insufficient accuracy and low deployment efficiency of existing detection models in complex scenarios, an improved YOLO11 domestic waste detection model is proposed. By introducing deformable convolution and a self-designed three-branch coordinate attention mechanism, an enhanced deformable convolution module is constructed, which is used to reconstruct C3k2 in the backbone network, significantly improving the model′s feature extraction capability for targets in complex background. In addition, a content-aware feature recombination operator is adopted to replace the upsampling in the neck network, enhancing the feature reconstruction effect. An exponential moving average sliding loss function is introduced to improve detection accuracy effectively and accelerate model convergence. Experiments on the optimized Huawei Cloud domestic waste dataset show that the improved model achieves 76.5% and 64.6% in mAP@0.5 and mAP@0.5:0.95 metrics, respectively, with an increase of 1.8% and 1.7% compared to the baseline model. Compared with other mainstream detection algorithms, the improved model has a parameter count of only 2.8 M, making it more suitable for mobile deployment.