In order to solve the problems of slow response and instability in the flight and grasping process of UAV with manipulator in complex environment, the ISMO method is applied to the control of UAV with manipulator for the first time. Firstly, according to the position and attitude relationship in space and Euler-Lagrange equation, the overall kinematics and dynamics model of UAV and 3-DOF manipulator is established to ensure the accuracy of the system. Secondly, after using the mathematical model to describe the relationship between variables, a complex simulation environment is built to simulate the whole sampling process. Finally, the ISMO control rate is designed for the whole control, which is proved by Lyapunov equation. Considering the influence of the global dynamic and static environment on the disturbance of the UAV position, attitude, manipulator and the increase of load after grasping. The simulation results show that its response speed and robustness are better than the traditional PID controller, which ensures the efficient and stable operation of the system.