Based on the Matlab/Simulink simulation software, the principle of the weak sine signal detection and the simulation experiment steps are analyzed in the HolmesDuffing system. The simulation results show that there is a large human error in the threshold value of the chaotic state directly from the phase diagram. Aiming at this problem, the Lyapunov exponent algorithm is proposed to quantitatively analyze the dynamic characteristics of the system and research the relationship between the maximal Lyapunov exponent and the system state. The further analysis of the critical region discovers that the LE exponent in the zero oscillation shows that the chaotic system is in the intermittent criticality.Finally,the weak signal with signal to noise ratio of -40 dB is tested to show that the Lyapunov exponent algorithm can judge the existence of weak signals more accurately, which proves the feasibility of the detection method.