In order to coordinate various control methods of AC/DC power systems and reduce the operation cost in steady state,a mathematical model is established to calculate the optimal power flow of AC/DC power systems.The proposed model,which is characterized by its nonlinear mixed integer programming considers not only the discrete characteristics of transformers（including converter transformers） ratio but also the shunt capacitor switching susceptance.The model is then solved by using the nonlinear primal-dual interior-point algorithm incorporating a quadratic penalty function to deal with the discrete control variables.Moreover,the introducing occasion of the quadratic penalty function and the valuating strategy of the penalty factors during the iteration are discussed.Calculation instances of two AC/DC power systems with 11 buses and 39 buses show that both the proposed optimal model and the algorithm can reliably converge to the enforceable and economic operation mode of the system.