In the contour measurement system based on machine vision, optical distortion exists in the imaging due to the uneven lens transmittance of the camera, which affects the measurement accuracy. In order to correct the optical distortion, the distortion mathematical model is firstly established, and then the distorted contour data is obtained by measuring the square calibration plate. According to the geometric characteristics of the distorted contour data and the linear constraint conditions, the parameters in the distortion model are calibrated by using LM optimization algorithm. The same contour measurement system was calibrated several times by experiments and the calibration results were compared with the existing methods. The experimental results show that the standard deviation of the proposed method is smaller than that of the existing algorithms, which indicates that the proposed method has better robustness.