In order to explore the difficulties in detecting fine cracks and the occurrence of breaks in the aerial view of UAVs, a network called ASE-Net is proposed based on the U-Net architecture. First, an improved VGG-16 is used as the encoder to extract the broken feature information. Second, multi-scale feature fusion block (MSFF) module and channel enhanced strip pooling (CESP) module are introduced at the minimum scale network layer. Finally, the ECA_X attention module is added to the decoding stage. The experimental results indicate that the model presented in this paper achieves a mIoU of 0.820 9, a mPA of 0.930 2, and a mPrecision of 0.865 1 on the self-constructed UAV aerial pavement breakage dataset. These results represent improvements of 15.97%, 12.72%, and 11.02% over the baseline U-Net, respectively. Ultimately, the model in this work has been demonstrated to exhibit better performance and generalization ability than other standard models utilizing the open-source dataset Crack500. The model can realize accurate detection of small cracks, potholes, and repairs on the road surface, effectively solving the fracture problem of crack detection, and enhancing the effect of pavement damage detection in large-size aerial images.