Aiming at the problem of measurement accuracy of the straightness of deep hole tubes, a measurement method based on the principle of optimized by the geometric approximation searching (GAS） algorithm is proposed. Firstly, the position of the center of each cross-section is obtained step by step by the measurement device. Secondly, the least squares (LS) algorithm is used to fit the center of each cross-section to locate the reference axis. Finally, the fitted axis and straightness obtained by the LS algorithm are taken as the initial parameters, and the GAS algorithm is used to carry out the optimized iterative calculation to obtain more accurate straightness error. At the same time, the measurement software developed by laboratory virtual instrument engineering workbench (LabVIEW） can monitor the data in real time. The experimental results show that compared with the single use of the LS algorithm, the measurement accuracy of the optimized iterative combination algorithm (LS+GAS) improved by 5.24% to 19.27%, and the uncertainty of the measurement results is reduced by an average of 24.82%. The algorithm of optimized iterative combination shows high accuracy and good robustness in fitting the axis, and combined with the self-propelled device, it can significantly improve the efficiency of straightness measurement.