A directly robust adaptive controller based on the radial basis function (RBF) neural network was proposed to solve the problem of the trajectory tracking for anti-sway control of an underacuated overhead and gantry crane with the parameter uncertainties and external disturbances. Firstly, the dynamic model of a double-pendulum crane was established by Lagrange equation of the second kind, and an ideal controller was designed for undisturbed conditions. Secondly, the RBF neural network was introduced to fit the ideal control output and its weight update rate was designed considering the effects of disturbances, after that the Lyapunov stability of the system was proved. Finally, a simulation test was carried out to verify the controller performance. The controller only takes the real-time position and speed information of the trolley as the control input, and no system parameter information is needed. The control accuracy depends on the fitting accuracy of the RBF neural network. The simulation results show that the controller proposed in this paper has excellent trajectory tracking and anti-sway performance and strong robustness.