Buildings in aerial images often have multiple orientations. The target detection algorithm based on the traditional convolutional neural network mostly uses the horizontal anchor frame as the detection frame, which has a low accuracy in detecting the building scene with multi orientation distribution. Therefore, this paper proposes a target detection algorithm based on CenterNet neural network, adds angle prediction branch on the basis of CenterNet model, and integrates the orientation angle information into the network. Aiming at the problem that few building angle features are extracted in the feature extraction stage of CenterNet model, asymmetric convolution is used to replace the original 3×3 convolution to enhance the feature extraction ability of neural network for rotating target angle information, and reduce the impact of angle periodicity on target detection by improving the loss function. The improved network can more accurately detect buildings with multi orientation distribution. In this paper, experimental tests are carried out on the data set built by ourselves. Under the same environment, the overall average precision is improved by 5.2% before and after the network improvement，including 74% for buildings with large orientation changes within the range of 10°~80° and 100°~170°. The average precision of buildings with small orientation changes within the range of 0°~10°, 80°~100° and 170°~180° has increased by 31%, effectively improving the average precision of buildings with multiple orientations.