In view of the visual limitations of the current camera attitude estimation methods. In order to realize the relative attitude estimation of wide-FOV( Fields of view camera), this paper used fisheye lens as visual sensor for attitude estimation. While fisheye imaging has the advantages of a wide-FOV, it is accompanied by serious nonlinear distortions, which leads to the problem of different distortion diffusion at different azimuths and distances. Therefore, this paper proposed a method to directly use the non-linear characteristics of the fisheye image to measure the relative pose of the camera. First, established the fisheye dataset kitti_FE; Secondly, used convolutional neural network for feature extraction and then combined with Long short-term memory network for bidirectional loop training to achieve the end-to-end output of the relative posture of the camera; Finally, the method of transfer learning was used to estimate the pose of the fisheye camera in the actual scene. Experiments show that the proposed method is 32%、29% and 25% higher than the camera pose estimation accuracy under the existing frameworks of CNN 、DeepVO and CNN-LSTM-VO-cons, respectively, and the proposed method is more stable under high-speed motion.