Bolts and rivets serve as essential fasteners in engineering applications, including transmission lines, railway transportation, bridges, and aircraft. However, exposure to external environmental factors makes them susceptible to various faults, such as missing pins, loose nuts, corrosion, and structural damage. Accurately detecting these faults is crucial for ensuring the safe and stable operation of transmission lines, railway systems, aircraft, and other related infrastructures. Leveraging large-scale data, deep learning-based bolt and rivet fault detection employs convolutional neural networks (CNNs) to automatically extract deep image features through hierarchical learning. By optimizing network parameters, these methods enhance feature extraction and generalization capabilities, yielding superior detection performance compared to traditional image processing techniques. This paper provides a comprehensive review of vision-based bolt and rivet fault detection research over the past decade. It begins by outlining common fault characteristics and the challenges associated with visual inspection. Next, deep learning-based detection approaches are categorized into three main types: two-stage algorithms, one-stage algorithms, and cascaded detection models. The paper then explores visual fault detection methods in key application scenarios, including line-type, box-type, and component-type bolts and rivets. Finally, it discusses challenges in machine vision-based fault detection, such as dataset limitations, sample annotation, and small target detection. By integrating existing deep learning technologies with the latest research advancements, this study presents an in-depth analysis of future development trends in deep learning-based bolt and rivet fault detection.