In recent years, unmanned aerial vehicles have been widely applied in the field of remote sensing image acquisition. However, in the task of target detection within remote sensing images, single-modal visible light images exhibit certain limitations, as their detection performance significantly degrades in dark environments or under adverse weather conditions such as haze. To address this issue, this paper proposes a dual-channel remote sensing image target detection network based on YOLO, named YOLO with dual-channel fusion and Mamba（YOLO-DFM）. This network integrates the Mamba architecture with the YOLO model and incorporates the dehazing module C2f with dynamic-range histogram self-attention (C2f_DHSA） to enhance the detection capability of visible light images. Additionally, we introduce a visible-infrared profound semantic fusion module (PSFM), which effectively fuses visible and infrared images, leveraging the complementary advantages of both modalities to improve the robustness and accuracy of remote sensing image target detection. For feature extraction and fusion, we propose a dual-convolution （feature multi-scale composite convolution，FMCConv） module, which combines the advantages of space-to-depth convolution (SPDConv） and deep robust feature downsampling (DRFD）. By employing an innovative feature fusion strategy and computational optimization techniques, this module enhances target detection accuracy in complex multi-scale backgrounds and is integrated into the YOLO-DFM network. We validate our proposed method on the vehicle detection in aerial imagery (VEDAI）, low-light visible-infrared paired (LLVIP） and DroneVehicle datasets. Experimental results demonstrate that, compared to the original YOLOv8, YOLO-DFM improves the mAP metric by 6.8%, 1.2% and 1.6%, respectively, significantly enhancing detection performance and confirming the model′s effectiveness. In conclusion, the YOLO-DFM network proposed in this paper provides an efficient and reliable solution for multi-modal remote sensing image target detection.