In order to solve the current moving object tracking algorithm in complex background model and target characteristics is not obvious, lead to the problem of insufficient algorithm tracking ability, this paper respectively from the feature point detection and the perspective of optical flow method, is proposed based on feature point detection and optical flow method of moving object tracking algorithm. First of all, according to the minimum image gradient matrix eigenvalue, by affine transformation, accurate feature points matching between frames, eliminate the false feature points and achieve the purpose of accurately detecting moving target feature points. Then, based on image pixel conservation principle, the deformation between two image assessments, establish image constraint equations, further precise tracking moving targets. Finally, based on the software development environment QTCreator algorithm, and system integration. Test results show that compared with the current motion target tracking technology, the algorithm has higher accuracy and stability.