Dangerous rocks and falling rocks around the railway intrude into the railway boundary, which will seriously endanger the life and property of passengers and the safety of railway traffic Aiming at the problems that traditional detection methods have high false detection in complex dynamic environment and low accuracy of small target recognition, a video-based deep learning method for railway rockfall intrusion detection is proposed. First, a hybrid attention module is incorporated into the network structure, which can enhance the network's ability to detect rockfalls similar to the background. Secondly, part of the network structure of YoloX is improved to a bidirectional feature pyramid network, which strengthens the mutual exchange of features at different levels and improves the ability to identify small targets. Simultaneously collect a large number of simulated rockfall data from different scenarios, build a simulated rockfall data set, and use the Mosaic data enhancement method in training to enhance the generalization ability of the method. The experimental results show that with the addition of improved modules, the identification accuracy of the method in this paper is continuously improved. Compared with various mainstream target detection methods, the highest identification accuracy is achieved, and the identification of different sizes targets is stable, which proves the application value of the algorithm in this paper in the actual railway scene.