To tackle challenges such as complex backgrounds, scale variability, and difficulty in detecting small objects in X-ray security inspection images, a lightweight contraband detection algorithm named LEM-YOLO is proposed, focusing on edge and multi-scale features. Firstly, a lightweight edge feature enhancement (LEFE) module is designed to strengthen edge feature extraction. Secondly, an efficient multi-level feature fusion pyramid network is developed, incorporating dynamic upsampling (Dysample) and the hierarchical scale feature pyramid network (HS-FPN) to enhance multi-scale feature fusion while reducing computational redundancy. Additionally, a dynamic feature encoding (DFE) module is used to preserve global information for small objects. Finally, Shape-IoU is employed as the bounding box regression loss function, concentrating on boundary shape and scale to improve localization accuracy. Experimental results on the public SIXray dataset demonstrate that LEM-YOLO achieves a mAP of 94.63%, a 2.56% increase over the original algorithm, while reducing model size by 50.67%, making it more suitable for contraband detection scenarios.