Aiming at the problems of low accuracy, high missed detection rate and insufficient real-time performance of railway track defect detection, this paper proposes a rail defect detection algorithm based on YOLO-FCA. First, the backbone network of YOLOv7 was replaced with the lightweight network of FasterNet, and the attention module of CloAttention was added to reduce the number of parameters and calculation load while improving the accuracy of defect detection. Secondly, a multi-scale adaptive feature fusion network (MS-ASFF) is proposed to obtain high-level semantic information and retain low-level detailed features to enhance the accuracy and robustness of model detection. Finally, the network pruning is carried out without affecting the accuracy, which makes the model more lightweight and greatly improves the detection speed of the model. Experiments on public data sets show that compared with the original YOLOv7 model, the mAP of YOLO-FCA is increased by 4.1%, reaching 80.7%, and the detection speed is increased by 38.5%, reaching 212.5 FPS. The experimental results show that YOLO-FCA can locate and detect rail defects efficiently and accurately.