Through the cooperation of coarse pointing assemble and fine pointing assemble, the compound-axis system solves the contradiction between large angle and high precision tracking in space optical communication and is an important means to realize high-speed space optical communication. However, the coordination of coarse pointing assemble and fine pointing assemble also brings complex decoupling problems to the compound-axis system, which reduces the robustness of the whole system. When tracking targets with fast-changing speed, it is easy to oscillate, and even make the system in an unstable state. In this paper, the tracking bandwidth, accuracy, and stability of the system are analyzed based on the compound axis system by Byrd diagram, and it is concluded that the decoupling process is the main reason that affects the stability of the compound axis system. A linear auto-disturbance rejection compound axis control decoupling scheme based on tracking differentiator is proposed to restrain the oscillation divergence that may occur in the process of decoupling while ensuring the accuracy of the system. The compound axis tracking experiment is carried out on the satellite orbit simulation platform. The experimental results show that in the case of orbit motion with a maximum angular acceleration of 0.32°/s2 and micro-vibration of the maximum angular acceleration of 2.26°/s2, the improved compound axis control can effectively improve the tracking stability of the system, and the final tracking accuracy is better than 1?μrad(3σ).