Seepage is a key factor that causes the loss of fine particles in soil-rock mixture fillers and leads to changes in soil structure and subgrade deformation and instability. In this paper, the self-developed particle loss test device was used to carry out the seepage test of soil-rock mixture fillers with different gradations. The change process of permeable quality, fine particle loss and settlement of fillers under seepage was monitored, and the evolution process of soil structure was analyzed. Particle Flow Code in 3D (PFC3D) was used to simulate the process of fine particle loss, and the dynamic change characteristics of soil porosity and filler particle size were analyzed. The results show that the permeability quality and fine particles loss rate can reflect the sensitivity of soil-rock mixture fillers structure to water. The lower the sensitivity of filler skeleton structure to water, the less the damage of the seepage to the structure, because the water can be discharged timely and effectively. The structural evolution process of soil-rock mixture fillers under seepage conditions can be divided into three stages, namely, stage of rapid loss of fine particles, skeleton remodeling stage, and relatively stable stage. The skeleton remodeling stage is the key stage to cause structural damage. In the stage of rapid loss of fine particles, the growth rate of per hour permeable quality is fast, a large number of fine particles are lost, and the settlement is small. In the skeleton remodeling stage, the change rate of per hour permeable quality slows down, and the loss of fine particles decreases. However, the reorganization of skeleton structure leads to obvious relative displacement of particles, and the settlement increases rapidly. In the relatively stable stage, the per hour permeable quality changes slowly and the settlement remains basically unchanged. The process of particle loss numerical simulation shows that the migration of fine particles causes the change of porosity of fillers, resulting in the change of soil structure. The results show that, among the studied fillers, the soil-rock mixture fillers with n=0.55 has the most stable skeleton structure and the weakest sensitivity to water.