To address the missed and false detection issues of YOLOv8 in identifying foreign object debris (FOD) on airport runways—caused by small object sizes, random spatial distribution, and significant scale variations among debris—this paper proposes AMMS-YOLOv8, an enhanced model incorporating prior masks specifically for small targets. In the backbone network, an isotropic edge detection operator is introduced to construct EIEStim, strengthening the model′s perception and preprocessing capabilities for subtle edges. Simultaneously, downsampling is replaced by an improved receptive field attention mechanism applied to the detection domain, forming LDFDS to enhance spatial awareness and preserve minute semantic information. Subsequently, the Neck layer is restructured to enable multi-scale feature aggregation, developing CCFPN to improve semantic perception of multi-scale debris. Finally, prior FOD mask features are embedded into the detection head and concatenated with deep features to create MSN-Head, thereby amplifying spatial perception. The model′s detection capability was validated using a self-built complex-scenario FOD dataset. On this dataset, AMMS-YOLOv8 achieved improvements of 1.8% and 1.7% in mAP50 and mAP50.95 respectively, with precision, recall, and F1-score reaching 0.971, 0.976, and 0.973—marking significant enhancements over the baseline network. Experimental results confirm the efficacy of these improvements. Furthermore, robustness and generalizability were evaluated through comparative experiments using a hybrid dataset (combining complex-scenario FOD data with FOD-A) and a complex transmission line FOD dataset, demonstrating performance gains across all metrics.