The typical characteristic of helicopter vibration environment is a mixed-spectrum excitation composed of low-level broadband random vibration superimposed with intense sinusoidal vibration. Traditional design methods for such inertial navigation vibration isolation systems heavily rely on engineering experience and often require multiple iterations, leading to low design efficiency.To address these issues, a novel vibration isolation system design method is proposed. Taking a fiber optic strapdown inertial navigation system (INS) as the research subject, the study first establishes the motion differential equations of the vibration isolation system. Based on the principle of motion decoupling for elastically supported rigid bodies, a structural design form of a triple-symmetry eight-point vibration isolation layout is elaborated. To tackle the challenge of directly analyzing system responses under such mixed-spectrum excitation, the fixed-frequency sinusoidal excitation is equivalently converted into narrow-band random vibration. Using the frequency response function substructure synthesis method, a stochastic dynamic analysis model of the vibration isolation system is constructed. Based on this model, the influence of variations in damper stiffness parameters on the root mean square (RMS) acceleration response of the isolation system is systematically analyzed. It is found that the system response exhibits a non-monotonic trend—first decreasing and then increasing—with increasing damper stiffness. Accordingly, the optimal stiffness parameters are determined based on the response minimization criterion. To validate the effectiveness of the proposed method, comparative experiments were conducted between the isolation systems designed using the proposed method and the traditional frequency-avoidance method. Experimental results show that the acceleration transmissibility curves along the X, Y, and Z axes of both systems are essentially consistent, verifying the effectiveness of the motion-decoupling-based eight-point isolation layout. Compared to the traditional method, the new method reduces the RMS acceleration responses by 26.9%, 24.4%, and 24.7% in the X, Y, and Z directions, respectively, significantly improving overall vibration isolation performance and demonstrating the rationality and engineering applicability of the proposed method.