In the image classification based on the spatial pyramid matching model, all the features in the image are treated equally when the histogram of visual words are constructed, without considering the influence factors of the different regions in the image. Obviously, the target area in image is more important than the background area. In order to avoid that the features of the nonimportant area in the image bring interference, this paper proposes an image classification method to optimize the spatial pyramid model. Firstly, a visual dictionary is constructed by using a clustering algorithm combined with a simulated annealing algorithm and a genetic algorithm. Then, a weighted visual word histogram is constructed using the visual attention mechanism. This method also takes the importance of classifying the images in each region of the image into account without losing the global information of the image. Finally, SVM is used to train and classify the representation vectors of images. Experimental results show that the proposed method can improve the accuracy of image classification.