Aiming at the problems of low efficiency and poor precision of visual track recognition in traditional UAV inspection, an algorithm for track detection based on LSD constrained cluster fitting and Kalman filter is proposed. Firstly, IPM algorithm was used to correct the Angle distortion caused by the lens Angle, and the track contour was detected by LSD algorithm. Under the constraint of track spacing, LSD results were clustered and the track lines were obtained by least square fitting. Then, a mathematical model is established according to the geometric characteristics of the orbit and the dynamics characteristics of UAV, and the track coordinate information is estimated by Kalman filter to ensure the stability and robustness of the algorithm. The trajectory images of multiple scenes collected by UAV are used as test samples, and the detection algorithm is compared with other algorithms. The experimental results show that the track recognition algorithm in this paper is better than other algorithms, and its track accuracy recognition rate reaches 92.49%, and the recognition rate reaches 23 frame/s, which meets the stability and real-time requirements of track detection.