Due to the large intra-class gap of remote sensing scene images, that is, the feature information of the same category of images is quite different, the accuracy of classification based only on feature information is not high, and the existing remote sensing scene image classification methods ignore the same common information of the same category. It can assist in image recognition. This paper proposes a remote sensing scene image classification method based on the fusion of common and characteristic information. First, the simple feature map and the complex feature map obtained by the shallower and deeper layers of the convolutional network are superimposed on the image, which can be considered as the feature map with concentrated attention of the image, and the handcrafted feature LBP of this feature map is extracted as the common information. It is then fused with the feature information extracted by the convolutional network and classified. In this paper, the SVM classifier whose hyperparameters are optimized by Bayesian optimization is used to achieve the best performance to eliminate the influence of the classifier on the experiment. Experiments on the two datasets UC Merced and AID verify that the classification accuracy reaches 98.80% and 96.06%, respectively, indicating that the method can effectively improve the accuracy of remote sensing scene images. It is of great significance in the fields of national defense, urban planning, and geological exploration.