The rapid expansion of the application range of unmanned systems makes the visual perception environment more complex and changeable, which makes it difficult for traditional visual control algorithms to effectively control visual sensors to obtain accurate visual perception images, thus affecting the stable operation of unmanned systems. Therefore, the research on intelligent visual control algorithms based on unmanned systems is proposed. The gray value of the visual perception image of unmanned system is transformed by Gamma curve nonlinear, and the contrast of the image is enhanced by the gray world method. Based on the processed image, the image moment is calculated, namely the space moment, the central moment and the normalized central moment, to describe the global and local characteristics of the image. According to the obtained visual perception information of the unmanned system, the intelligent visual control framework is built. Obtain the desired image feature matrix, extract the current moment image feature matrix, and nonlinear map the camera angle through the extreme learning machine based on the improved firefly algorithm, so as to obtain the intelligent vision control law, so as to eliminate the visual perception image error and realize the effective control of intelligent vision. The experimental results show that under the background of different experimental groups, the minimum average time of visual control obtained by the proposed algorithm reaches 1 s, and the minimum average error of visual control reaches 0.12%, which fully confirms the better application performance of the proposed algorithm.