There are problems such as feature imbalance and insufficient feature fusion in the visible and infrared image fusion pedestrian detection algorithm. To address the above problems, we propose a multispectral pedestrian detection network MIFNet with phased feature fusion, a dual-stream network that handles both visible and infrared inputs, an intermodal information fusion module that changes the structure of the network to reduce the impact of feature imbalance, and an extraction-injection structure that automatically learns how to extract multimodal global information during the process of feature extraction and injects it into the visible and infrared features efficiently, which improves the robustness of the network and feature fusion effect. The feature enhancement fusion module is designed and embedded to enhance the unique information of the two modalities to further improve the feature fusion effect. The experimental results show that the leakage rate of the algorithm is only 9.74%, which is 6% lower than that of the baseline algorithm, effectively improving the detection performance of the algorithm.